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SECOND GUESSING PROGRAM EVALUATION: 
THE NEED FOR CONTEXT 


More and more, federal departmental program evaluations 
are coming under external scrutiny. Among the factors 
nfluencing this development have been the Access to 


Information legislation, the Ministerial Task Force on 
Program Review, the increased interest in program 
performance on the part of Parliamentary Committees, the 
emerging Increased Ministerial Authority and Accountability 
régime (IMAA) and the continuing value-for-money auditing 
by the Auditor General. While all these developments are 
indicative of the maturing of program evaluation in the 
federal government, the increased scrutiny has caused some 
legitimate concern in the evaluation community. 


These external users of departmental evaluation reports, users 
not anticipated at the time the studies were done, have 
frequently been critical that the study (a) did not address 
the issues of interest to them, (b) were not completed when 
they wanted them or (c) did not use a sufficiently rigorous 
methodology. Since an evaluation can never be all things to 
all people, some of these criticisms are unavoidable. But with 
a little extra work on the part of the evaluator, I think a 
significant amount of the criticism can be avoided. 


Getting more exposure (which is one response to all the talk 
about accountability), requires more care to document in the 
report what the context of the evaluation was. The report 
should, clearly and succinctly, answer such questions as: 


e Was the evaluation study the main (only) information used 
by the department to review the program, or was it part 
of a broader process of reconsideration? 


Often the evaluation study per se is only a part of the 
program review. In such cases, the evaluation report on 
its own may not - and it is not intended to - reflect all 
the issues addressed and evidence considered. Accordingly, 
criticisms may be either inappropriate or might more 
properly apply to the overall process rather than to the 
evaluation study. 


Canada 


e¢ Why was the evaluation undertaken? What were the events 
surrounding the program that led the department to 
evaluate it when it did? 


This information would put in context the focus of the 
ensuing study. As an aside, explaining that the study was 
done because it was so scheduled, while satisfying internal 
variance reporting requirements, would not, it seems to 
me, give an outside reader much of a rationale! 


e Why were the issues addressed chosen at this time and, 
more importantly, why were other issues not addressed? 


The import of this information is evident. Without it, and 
with hindsight, it is always “obvious” that certain other 
issues should have been addressed. 


e Why was the timing for the completion of the study and 
the particular degree of rigor chosen? 


Sometimes a quicker, less rigorous study is exactly what 
is called for to meet impending decision information 
needs. At other times, given the nature of the issues 
addressed, a longer, more rigorous study is required. With- 
out this explanation in the report, other readers are likely 
to be critical of either choice. 


e Why was the particular level and form of documentation 
of the study chosen? 


Sometimes a short executive summary may be the main 
document, backed up with working papers, data and notes 
in a form not easily understood by an outsider. If little 
challenge to the report is expected, or if the evaluation 
study is only part of the information used by the depart- 
ment to reconsider the program, this may be fine. In other 
cases, more thorough documentation of the study may 
be needed. 


Some reports do cover these topics, but in most cases I have 
seen, it is done in a rather perfunctory manner - something 
you have to do rather than something you want to do. It 
seems to me that if a report is written up with a view to fore- 
stalling future second-guessing of what was done in the evalu- 
ation, it might be done with more vigor and care. Or, more 
positively, if an evaluation study report is written more as 
a departmental management report than as a research report 
on a program, it would attract somewhat less unjustified 
criticism. It is, after all, the only chance the evaluator will 
get, in many cases, to explain and defend the evaluation from 
the Monday morning quarterbacks that abound. 


John Mayne 


John Mayne is the Assistant Director, Policy Division, 
Program Evaluation Branch, OCG. 


EFFECTIVENESS: REPORTING AND AUDITING I 
THE PUBLIC SECTOR 1 


A summary of a commentary, by Guy Leclerc, Depu 
Comptroller General, Program Evaluation, OCG to t 
published in the Canadian Journal of Program Evaluatio: 
on the Report of the Canadian Comprehensive Auditir 
Foundation entitled, Effectiveness: Reporting and Auditir 
in the Public Sector. These comments do not constitute th 
“official position” of the Office of the Comptroller Gener 
on the report. | 


In Canada, public expenditures amount to some 40 per cel 
of the gross domestic product. The federal government 

a major social and economic agent and is a major interven 
in the market place. Accountability for these responsibil 
ties has traditionally been based on the government 
financial statements (the Public Accounts and the Estimates 
economic statements (the National Accounts and tl 
Budget), on the audits by the Office of the Auditor Gener: 
and, recently, on departmental program evaluation. 


By international standards, Canada is well served by suc 
vehicles for accountability. The focus on probity and integri 
issues, while justified, has distracted attention fro! 
discussion, in an accountability context, of the substanti' 
activities of government. Probity and integrity issues a 
easier to circumscribe than those related to responsibility f¢ 
the effectiveness of public expenditures. 


Accountability for Effectiveness 


A major factor contributing to the elusiveness « 
effectiveness issues is the nebulous interface between tl 
political and the technical. Objectives pursued at the politic 
level are sometimes unstated and are often only loose 
related to those pursued at the operational level. In additio! 
the analytical processes of ex-post evaluation or comprehe! 
sive audit, while quite adequate in many respects, may b 
as instruments of accountability, somewhat cumbersome, 1! 
timed and unenlightening on the real dynamics, political ar 
substantive, of policy areas. 


Management Representations on Effectiveness 


The Canadian Comprehensive Auditing Foundation (CCA! 
report on Effectiveness: Reporting and Auditing in the Pubi 
Sector is a landmark in the debate over effectiveness ar 
accountability relationships between public sector manag 
ment and their “governing bodies”, such as the feder 
Parliament. The Foundation has shown great initiative 
eliciting views from an impressive array of concernt 
Canadians, elected people, senior administrators, senior ci 
servants, and professionals. The report succeeds in pene 
much coherence to a difficult area. 


The CCAF report says that the way to deal with effectivene 
in reporting to Parliament is through a set of ‘“manageme 
representations” designed to help Parliament form jud 
ments about effectiveness. I agree with the Foundation th 
management, rather than the external auditor, shou 
evaluate or assess performance and report on it. The audit 
should be concerned with providing opinions on t 
adequacy and balance of these reports. 


b me, management representations on effectiveness are an 
yproach that extends the usefulness of the 1977 Treasury 
oard policy initiative on departmental program evaluation. 
| e management representations approach may offer more 
lequent and flexible channels of accountability. Manage- 
lent representations could be prepared annually, tailor-made 
br a variety of stakeholders, although the CCAF report 
fgues that representations on the full range of a 
partment’ s activities are No to be spread over longer 
eriods of time. 


he structure and content of such representations are more 
roblematic. Program evaluation has recently been exploited 
Ss a source of accountability statements in Part III of the 
stimates. Similarly, program evaluation could feed any 
pproach based on making representations on program 
erformance. At this point, however, although we know how 
lanagement representations as a tool of accountability 
ould flow, we know very little about their content. The 
yrmulation of reasonably structured representations is going 
D be a developmental task not unlike the development, years 
go, of financial statements. 


ome Lessons of Federal Evaluation 


ederal program evaluation has learned several lessons about 
ffectiveness measurement and reporting which are relevant 
ere. At a recent hearing of the Standing Senate Committee 
In National Finance, Mr. J.A. Macdonald, Acting 
‘omptroller General of Canada, offered the following: 


Single statements and single measures of program 
bjectives, results, and effectiveness almost always are 
nadequate. Any government concerned with value-for- 
honey must seek to achieve as many objectives as possible 


| ell. This adds an additional dimension to the effectiveness 
)f expenditures in these related programs. 


‘One of the ways we are responding to this difficulty is to 
rticulate a hierarchy of results for programs, including: 
a) broad, sometimes government-wide, impacts and effects; 
b) more immediate benefits to program clients; and, 
c) specific program outputs produced by a delivery unit 
vithin a department. In this way, we believe we can have 
easonably reliable measures of the more operational 
utputs. At the same time, we will still have a relevant context 
>f related effects to provide a solid information base on 
which to make judgments of effectiveness in terms of overall 
rogram and governmental objectives. 


“To measure effectiveness in this sense, it is important not 
o limit ourselves to quantitative indicators. Accordingly, we 
ire also relying on qualitative information derived from the 
analysis of specific performance issues. This approach 
equires more interpretation and judgment by users of the 
ffectiveness information, but avoids the almost inevitable 
dias and oversimplification of uninterpreted quantitative 
measures. 


“In addition, we are focusing less on conclusions concern- 
ing program effectiveness and more on relevant and reliable 
information on effectiveness. This provides for the various 
uses of such information - including Ministers, senior public 
servants, Parliamentarians, and interested groups and 
individuals - to make their own judgments about the need 
for and value of programs, all from their respective vantage 
points.” 


Audit Assurance 


The development of audit assurance for management 
representations will also be a challenge. In line with its 
tradition, the audit profession may have to develop detailed 
generic standards in this area unless it wishes to relax its 
tradition towards a more pragmatic approach. In addition, 
it has been suggested that auditors develop the structure of 
the management representations so that they would be 
auditable. In my view, auditors should probably not partici- 
pate in the formulation of representations that they will later 
be called upon to agree or disagree with. 


Another problem is that managers have to be concerned not 
only with the actual delivery of programs but also with the 
manner in which the superordinate objectives of the govern- 
ment have been respected. Auditors may not be ready to deal 
with macro-economics, social development issues and with 
being forced to reconcile all the contradictions of the political 
process. 


Conclusion 


The central merit of the Foundation’s report is the raising 
of program effectiveness as a major concern for many of 
the players and stakeholders concerned with public sector 
programs. The accountability approach of the CCAF flows 
from review processes already in existence. The best manage- 
ment representations can build only on a culture of effec- 
tiveness reviews. Indeed, a good program evaluation culture 
is a precondition, and probably not the only one, for good 
management representations on effectiveness. 


Guy Leclerc 


REPORT ON THE SECOND ANNUAL 
CES-NCC WORKSHOP 


The National Capital Chapter of the Canadian Evaluation 
Society (CES-NCC) hosted its Second Annual One-Day 
Workshop entitled “Evaluation: Forging the Links” on 
November 4, 1987 at the Ottawa Congress Centre. Partici- 
pation at the Workshop jumped by almost 35 per cent over 
last year, from 150 to over 200. In addition to continuing 
representation from the federal government, the consulting 
sector and the academic community, the Workshop attracted 
new representation from the Government of Ontario and the 
student body at the University of Ottawa. 


The day began with a thought-provoking keynote address 
by Mrs. Georgina Wyman, Deputy Minister, Supply and 
Services Canada, in which she paralleled the process of 
program evaluation with “thinking in the time streams”. As 
Mrs. Wyman expressed it “. . evaluation looks at a program 
in historical terms, but very much with a view to the future. . . 
(with) the idea of reducing those obstacles that cannot be 
removed. . .a step by step progress”. 


Concurrent sessions were then held on the linkages between 
evaluation and policy, audit and public reporting, followed 
by a stimulating luncheon address by Mr. Timothy Plumptre, 
Principal, William M. Mercer Ltd. Borrowing from his forth- 
coming book, Beyond the Bottom Line, he provided a num- 
ber of insights into the need to manage the public sector’s 
human resources better. Several additional concurrent 
sessions on the linkages between evaluation and Parliament, 
Increased Ministerial Accountability and Authority TMAA) 
and performance measurement followed. 


An invigorating closing address was delivered by 
Mr. J. Andrew Macdonald, Acting Comptroller General of 
Canada, who, reflecting on the recent resurgence in evalua- 
tion activity at the federal level, observed that the function 
was “reaching for the next rung”. Mr. Macdonald noted that 
the IMAA environment was stimulating the growth in de- 
mand for evaluation and that the Office of the Comptroller 
General would be playing a stronger, more active role in coor- 
dinating the evaluation interests of central policy-makers with 
the evaluation activities of line departments. 


Given the success of “Evaluation: Forging the Links”, the 
CES-NCC had decided to hold a Third Annual One-Day 
Workshop on Wednesday, November 2, 1988. Plan to be 
there. 


OECD MEETING ON EVALUATING PROGRAMS THAT 
PROMOTE TECHNOLOGICAL INNOVATION 


The following article was submitted by Alain Barbarie, 
Senior Policy Analyst, Program Evaluation Branch (OCG), 
presently on an Interchange Canada assignment with the 
Library of Parliament. 


The Technical Co-operation Service and the Directorate for 
Science, Technology and Industry of the Organization for 
Economic Co-Operation and Development (OECD) held a 
meeting in Paris, December 16-18, 1987, on the evaluation 
of programs promoting technological innovation. This was 
the third such meeting in the context of a project on 
Innovation Policy Implementation and Management. The 
first meeting, which took place in October 1985 in Paris, was 
devoted to Technological Innovation Policy. The second, held 
at the end of 1986, was on the promotion by public 
authorities of a technologically innovative environment. 


This third meeting was to give officials from government 
agencies concerned with evaluating programs and policies 
promoting technological innovation the opportunity to 
examine and discuss the evaluation approaches and metho- 
dologies of other Member countries. Participating were 
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people who ask for evaluation, people who evaluate, an 
people who are evaluated. A preparatory expert brair 
storming session identified some key issues, which wel 
explored at the meeting. 


Seven commissioned papers described relevant issues 
Member countries. I presented a paper on the Canadia 
experience entitled, “Evaluation of Programs Promotin 
Technological Innovation in Canada’’*. The purpose of thes 
papers was to generate a discussion to elucidate specifi 
questions, such as: 


- How much importance is given to the evaluation c 
innovation programs in the policy-making process? 


— Who is requesting such evaluations? 
— What institutions are in charge of such evaluations? 
- What is the reaction of the bodies or persons evaluated 


- What main issues should be tackled in order to increas 
the contribution of evaluation to policy-making? 


- What general lessons can be derived, which may be € 
interest to OECD countries in general? 


Following the presentation of the papers, the approximate! 
50 participants were divided into three working groups. Th 
gave everyone an opportunity to engage in in-depth discu: 
sion, in parallel, on the principles that should be observe 
in evaluating government-financed technology innovatio 
programs. 


On the basis of the papers presented and the discussions hel 
during the meeting, a report on approaches and methods wi 
be prepared for use as a working instrument by practitio1 
ers in charge of devising evaluation schemes for innovatio 
policies and programs. | 


Alain Barbar 


*This paper is available from the Program Evaluation 
Branch (OCG). To obtain a copy contact the Editor of 
the PEB Newsletter. 


Comings and Goings. . . 


e A farewell note to Céline Leblanc, Senior 
Evaluation Analyst, Program Evaluation Branch, 
OCG, who has recently joined the Program 
Evaluation Division at Transport Canada. 


+ PEB welcomes Jeanne Langford as an 
Evaluation Analyst with the Departmental 
Operations Division. Jeanne was formerly with. 
the Management Practices Branch of the OCG. 


CG PRACTICUM SESSIONS - 1987-88 SEASON 


e first session, on Information for Accountability, was 
Hd November 4, 1987 and featured the perspective of 
easury Board Policy Centres. 


lian Darling, Deputy Secretary of the Program Branch, 
leasury Board Secretariat, confirmed that the Increased 
linisterial Authority and Accountability regime (IMAA) i 1S 
foster a climate more conducive to productive manage- 
ent. He stressed the cooperation needed between depart- 
lents and central agencies to create the necessary change 
the cultural environment. He challenged departments to 
vy increased attention to accountability. In outlining IMAA 
hprovement initiatives, he related them to the constraints 
| good management articulated by the Auditor General 
1983. 


Indy Macdonald, acting Comptroller General, stressed that 
tcountability based on improved operational planning 
ameworks (OPFs) and adequate management control 
stems is fundamental to IMAA. He indicated that depart- 
ental program evaluation experience in identifying, 
leasuring and reporting on program results and in develop- 
lg evaluation frameworks is useful in: a) improving OPFs 
qd developing accountability frameworks; b) measuring and 
ocumenting evidence on the key program performance 
lsues; and c) reporting program performance information 
departmental management representations to Treasury 
loard. 


his first session clearly met its objective to communicate 

AA status, specifically emphasizing increased delegations, 
e simplification of processes and expectations for 
anagement representations. 


the second session, entitled Deputy Ministers, was held on 
anuary 3, 1988 and featured three deputy ministers sharing 
ipproaches, experiences and intentions about IMAA and the 
formation implications for managing their department and 
naking management representations. 


Louis Huneault, Deputy Minister of Revenue Canada - 
[ustoms and Excise, began the session by relaying some 
sssons learned from experience with negotiating a 
Memorandum of Understanding (MOU) under IMAA 
lecision. He mentioned that IMAA should be treated as an 
/pportunity to delegate and simplify throughout the whole 
brganization. The decision to embark should be based on 
olid and thorough homework and made only if there is real 
management commitment. In elaborating the organizational 
dvantage of delegation and simplification, he cautioned that 
ccountability i is the cornerstone upon which all else rests. 
Mr. Huneault felt that the staff functions of planning, 
ontrolling, reporting, internal audit and program evalua- 
ion must become active participants in the gathering, 


inalyzing and evaluation of information for management. 
ad 
4 
Gaétan Lussier, Deputy Minister/Chairman of Employment 
ind Immigration Canada described how the IMAA philos- 
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>phy was consistent with the management philosophy under 
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which his Department had been working. He described the 
planning and accountability process as being composed of 
mission statement; priority setting; operational plan; and 
follow-up (reporting, auditing and evaluating), with the OPF 
being the basis for measuring achievements against 
objectives. 


The approach of a common service agency towards IMAA 
was addressed by Georgina Wyman, Deputy Minister of 
Supply and Services. Proposals from other departments 
seeking increased or complete delegation in the common 
service area will be assessed in terms of trade-offs involving 
levels of service and cost to government. She emphasized that 
opting out of common services merely for philosophical 
reasons will be resisted. A focal point has been established 
in DSS and task forces are studying strategies and improve- 
ment opportunities. As accountability capability is 
fundamental, improvements in planning, audit, evaluation 
and contract quality assurance are being consolidated. 


The third session took place March 4, 1988, from 8:30 to 
12:00 at the National Library Theater, and dealt with Trends 
in Financial Information for Accountability. This session 
focussed on the expectations of the central agencies, 
parliamentarians and external users. 


On 7 April 1988, the final session in this Practicum series 
will be held on the topic “Accounting for Program Results 
and Performance”. The speakers will be: Guy Leclerc, Deputy 
Comptroller General, Barry Lacombe, Assistant Secretary, 
Program Branch, TBS, Ken Wyman, Executive Director, Un- 
employment Insurance Program, CEIC, and Mike Binder, 
Assistant Deputy Minister, Department of Communications. 
The session will focus on how information for accountabil- 
ity for program results and performance can be sensibly 
developed. 


For registration please contact Monique Thériault at 
957-7024. 


AN OCG WORKSHOP IN QUEBEC CITY 
ON SURVEY METHODS AND QUESTIONNAIRE 
DESIGN 


From January 20 to 22 the Office of the Comptroller General 
presented, for the first time, a professional development 
workshop for federal and Quebec government evaluators (in 
Quebec City) in collaboration with the Government of 
Quebec and Statistics Canada. The workshop dealt with 
survey methods and questionnaire design. The participants 
- 24 in all, from 8 federal departments and 14 provincial 
departments — agreed that the workshop was a great success. 


The workshop involved participants and resource people in 
many spirited discussions about survey design and 
implementation in the context of evaluation studies. It also 
gave participants from both levels of government an 
opportunity to share their experiences in evaluation. 


CIDA INTERNATIONAL SEMINAR: 
EVALUATION OF AID ASSISTANCE: 
THE YEARS AHEAD 


How do we ensure we learn from past experience? Can joint 
donor-recipient evaluations be undertaken? How can we 
improve donor coordination in the area of aid evaluation? 


In September 1987, the Canadian International Development 
Agency’s (CIDA) Program Evaluation Division hosted an 
international seminar entitled, ‘Evaluation of Aid 
Assistance: The Years Ahead”, to address these important 
questions. 


The seminar, held in the Nation’s Capital and in Val Morin, 
Quebec, attracted senior evaluation representatives from 
major bilateral donors, the Development Assistance 
Committee of the OECD and important international donors 
and financial institutions. In addition, Evaluation Officers 
from CIDA’s Program Evaluation Division and Operational 
Branches and members of the Office of the Comptroller 
General of Canada contributed the practical perspective of 
evaluation managers. 


The seminar focused on the challenging issues of evaluation 
follow-up and feedback, joint donor-recipient evaluation and 
information exchange among donors. Background papers 
prepared by CIDA provoked lively discussions among 
participants. 


Feedback from participants indicates that the seminar was 
highly successful and has contributed greatly to sharing 
evaluation experience and information among donors. The 
Expert Group on Aid Evaluation, a special group within the 
Development Assistance Committee (DAC) of the OECD, 
plans to follow up on substantive issues raised during the 
seminar. For example, CIDA will continue the development 
of a computerized information system on donors’ evalua- 
tion reports. The World Bank also committed itself to 
hosting a seminar to share its experiences with integrated 
rural development projects. These and other initiatives 


should help ensure that the momentum achieved at the 


Seminar is continued and that practical results are achieved. 


A formal conference report will be available during the first 
quarter of this year. 


Program Evaluation Division, Policy Branch, CIDA. 


Do you have an article of interest on program evalua- 
tion that you wish to submit for publication? The 


Newsletter welcomes all contributions from the evalu- 
ation community. For further information, please call 
the Editor, Paulette Somerville at (613) 957-7167. 


NEWS BRIEFS 


¢ On November 30 and December 1, 1987, 
Miss Marie-Odile Tesnière from the Agence nationale 
pour l’emploi (ANPE) of France met with Céline Leblanc, 
Program Evaluation Branch (OCG), Carole Viau, Internal 
Audit (OCG) and Louise Trépanier, Program Branch 
(TB), and the staff of the Program Evaluation Branch of 
the Department of Employment and Immigration. 
Discussions focussed on the establishment of an 
evaluation process within ANPE, the role and place of 


evaluation and the problems that might arise followin 
its implementation. Methodology, data banks and othe 
issues relating to employment were also discussed. 


Maryland, visited Ottawa. Dr. Schick is conducting | 
study for the OECD on the use of performance an 


process. During his visit, arranged by the Progran 
Evaluation Branch of the OCG, he met with severa 
TBS/OCG officials and with evaluation director 
(David Black - DOC, Ian Dawson - EAC ang 
Norman Steinberg - TC). | 


On January 15, 1988, Guy Leclerc, Deputy Comptrolle 
General, Program Evaluation Branch, was the invite 
speaker at an Ontario Management Board of Cabine 
seminar on the subject of government activity reviews 
Mr. Leclerc gave the audience an historical description 0 
how the Federal government’ s evaluation function wa 
developed and offered his views on how evaluation 1 
better and under what conditions. | 


An Information Exchange Seminar on Evaluation in R&LI 
Programs was held November 19, 1987. Mr. Joe Freamo 
Director of Program Evaluation at NRC, made som 
observations on the recent trends in R&D evaluation i 
European countries, based on his participation at th 

International Conference on R&D Evaluation, which wa 
held last Summer in Helsinki, Finland. Mr. Don Bennett 
Director of Program Evaluation at EMR, focussed hi 
presentation on his departmental R&D evaluatioi 
experience and the lessons learned. The session wa 

organized and animated by Toru Uno, Progran 

Evaluation Branch, OCG. Guy Leclerc, opened t 

session with comments on the recent Federal evaluatioi 
activities in the field of R&D. 


CANADIAN EVALUATION SOCIETY | 
1988 CONFERENCE 1 
DELTA HOTEL, MONTREAL, QUEBEC { 
MAY 16-18, 1988 


The general theme of this year’s conference will be 
“Evaluation with Limited Resources”. The conference — 
will focus on evaluation under non-optimal conditions. 
Particular attention will be given to effective strategies © 
for dealing with resource shortages: human, financial, 


information, and time. 


Plenary speakers, panel discussions, round tables, and — 
workshops have been arranged to focus on various — 
facets of the conference theme. One of the featured 
speakers will be Eleanor Chelimsky, Director of. 
Program Evaluation and Methodology, United States — 
General Accounting Office. 


Complete information concerning the conference 
program and on how to register can be obtained from — 
Natalie Kishchuk, Canadian Workplace Automation ~ 
Research Centre, Department of Communications, 
1575 Chomeday Blvd., Laval, Québec, H7V 2X2 
(tel: 514-682-3400) 
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B PRIORITY AREAS FOR 1988-89 


th year we summarize in the Newsletter the OCQ’s work 
orities in the area of program evaluation. Like most 
ices, we re-examine our priorities each fiscal year. 
is year, the principal vehicle for doing so has been the 
vernment Program Evaluation Plan (GPEP). We also 
ew the status of the evaluation function in those depart- 
nts covered by the GPEP. Through these activities and 
associated consultation with central agencies, we have 
ntified a number of specific actions that we will take this 
i. Two areas deserve particular attention. 


st, the coordination of evaluation planning needs improve- 
nt if it is to serve the internal management requirements 
departments and also serve the more specialized program 
iew interests of central agencies. Without such improve- 
nt, duplicate program evaluation procedures are inevita- 
pThey are also contrary to IMAA’s approach to manage- 
en nt. We are aware that this could be seen as “serving two 
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masters”. However, responding to the concerns that arise in 
administering a department’s programs and serving the 
interests of Cabinet as a collectivity are conventional respon- 
sibilities of Deputy Ministers. The Deputy Minister clearly 
remains the “client” for the products of the departmental 
evaluation unit. Harmonizing evaluation priorities would 
ensure that adequate plans are submitted to DMs for their 
consideration and that these plans do not duplicate effort. 


Improving this coordination means at least three things. 
First, as the evaluation focal point among central agencies, 
the OCG must stay abreast of all relevant central program 
review priorities and convey these to departments. Second, 
central agencies will need to articulate any interest they might 
have in particular reviews and help plan for the associated 
evaluation requirements. The negotiation of IMAA 
Memoranda of Understanding is one effective way to do so. 
Finally, departments must consider central agency require- 


ments for information on program performance when they 


plan their evaluations. 


The OCG is adjusting its activities to support this approach . 
better. The newly reorganized operations group in the 
Program Evaluation Branch will seek to harmonize central 
requirements with departmental issue and study priorities. 
It is arranging for better use of individual evaluations in 
support of broader interdepartmental reviews. It will pursue 
an improved meshing of expenditure program components, 
user fees units, and regulatory programs by modifying the 
OPF and in other ways. 


Second, departments, in the aggregate, must improve their 
capacity and readiness to evaluate their programs in a manner 
seen as credible by users. While several departments are 
producing credible and adequate studies, fewer have adequate 
coverage of their program base. To meet the enhanced 
accountability for program performance required by IMAA 
and related management procedures, departments will have 


Canada 


to improve overall coverage and quality. This will be pursued 
directly with individual departments, and through a special 
study on the productivity and resourcing of program 
evaluation. 


We believe the OCG can play a role in enhancing credibility 
and utility of departmental evaluations by conveying to 
central users of these findings its views of their salience and 
reliability. In support of this and the coordination role noted 
above, we will be tracking more closely evaluation issues 
addressed, findings, and departmental actions. 


In addition and complementary to these two general initia- 
tives, we will be continuing the work on IMAA, particularly 
for those departrhents where Memoranda of Understanding 
are being negotiated. This process does require substantial 
involvement of departmental evaluation as well as Program 
Evaluation Branch staff in the OCG. The two sectoral areas 
of regulatory programming and science and technology 
programming remain a particular concern, but not a change 
from last year. 


Finally, our mainstream activities of assuring the integra- 
tion of program evaluation policy into broader central 
management policy, development of evaluation methodol- 
ogy and maintenance of a core program of seminars 
and workshops remains important elements of our 
responsibilities. 


EVALUATION UTILIZATION PROJECT 


From time to time, the OCG has undertaken small projects 
to document the use made of program evaluation studies. 
We have used the results in marketing evaluation as a worth- 
while management process. Research was another motiva- 
tion. Studying the characteristics of useful evaluations, we 
felt, might reveal procedural and methodological lessons for 
future evaluations. We also needed examples for our training 
seminars and workshops. 


Federal departments and agencies have completed well over 
400 evaluations in the past five years. Some departments 
monitor the consequences of evaluations, sometimes 
publishing the results in annual reports and Part III of the 
Estimates. But there is no current overall documentation of 
the uses made of evaluation. Even knowledge about 
interesting individual studies is fragmented. 


The OCG now has an important new reason for updating 
its information on utilization. Rather than initiating new 
evaluations demands, central agencies, in carrying out their 
program review responsibilities, are hoping to make greater 
use of evaluation findings and information on resulting 
program improvements. 


Two parallel studies are examining the use of evaluation in 
the regulation and science and technology (S&T) sectors. 
These projects will allow us to demonstrate and assess 
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the extent to which program evaluation has been a relev. q 
credible and timely input to departmental and governme À 
decision-making in these areas. They may also serve (| 
identify opportunities and requirements for revising gui i 
lines about such evaluations that the OCG has issih. 
previously. 


There are three major lines of enquiry in these studies. Fi | 
a synthesis of the literature, building on past bibliograpAl: 
work, is being conducted. Second, the documentatifl 
and analysis of approximately 70 S&T and 30 regulatdl! 
evaluation studies, completed since 1984, is under way. Aff! 
third, two follow-up surveys of the results and utilization 
S&T and regulatory evaluations are to be completed. 


Beyond this sector work, we will update our data @ 
completed evaluations in major operational departme vi 
using a keyword classification. Studies will be described | 
leading to program redesign, savings/expansion, deliveff 
improvements or other classes of results. This will accom 
modate some broad summaries of the uses of evaluation | 
respond to the growing demand for such information. The | 
activities will also be coordinated with a project on the ul 
of evaluation being considered by the National Capit 
Chapter of the CES. 


In all these projects, attribution and subjectivity can | 
problems. Rarely will evaluation be the single cause ¢ 
program change and no successful study will be beyor 
criticism. Moreover, what worked for one project may n | 
necessarily work for others. Nevertheless, we need a bettél 
understanding of how evaluation and its findings are linke | 
to practical improvements in programs and their delivery & | 
well as a broader base of information for judging the valt | 
of expenditures on evaluation. Some of the results of the | 
projects may be available at the CES’s local chapte | 
November workshop. Those readers who can’t wait or wh 
wish to get involved can contact John Batchelor in PE 
at 957-7179. 1 
D | 
Project work on S&T and regulatory programs is expected 
to be completed this Fall. For further information, pleas 
contact David Moynagh (tel: 957-7174) about S&T program 
or Fred O’Riordan (tel: 957-7182) about regulatory program 


| 
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DEPARTMENTAL PERSPECTIVE 


APPLYING PROJECT MANAGEMENT 
TO EVALUATIONS 


Sound management practices can ensure that program evale 
ations meet their objectives and are completed in a timel 
manner. More and more, organizations are turning to forma 
project management approaches to help ensure such succes 
Having had first-hand experience in project management il 
various R&D organizations, I feel that we in the progran 
evaluation community would do well to consider applying 
formal project management to our studies. 


ations produce reports to influence organizations 
d of producing products to meet market wants, as is 
Jin project management situations. However, evalua- 
nd production are similar in other aspects. They are: 


ique, one-shot activities; 
hnically complex, usually requiring a team approach; 


used on an explicitly stated set of purposes or 
jectives; 


atively short-term activities lasting, perhaps, a year 
SO; 


lisible into substantial work components; and 


ct Management evolved in the early 1960s from Systems 
gement and it offers the following strengths: 


ghly results oriented; 
mergistic, owing largely to its multi-disciplinary teams; 


‘lies on communication, particularly written 
ymmunication/documentation within the project team, 
ad 

quires good people management, motivating and 
bcusing project staff on specific goals within the larger 
roject setting. 


yarticular relevance to evaluations is the attention project 

lagement pays to planning and controlling. As we can 

ppreciate, front-end planning can save a lot of grief later 

an evaluation, when it is difficult to overcome 

blems.The three elements in project planning (activities, 

urces, timing) receive great attention and various tools 

readily available to ensure that planning is thorough, 

istic, and sensitive to potential issues and problems 

| contingency planning). For example, project planning 
irts, one frequently used tool, help us identify the 
rarchy of activities down to the level of detailed subtasks 
1 then to organize activities into meaningful groupings. 

inning charts also enable us to identify resource require- 
ts, make time estimates and determine accountability for 
ivities. The scheduling aspects of project planning is facili- 
ed using (computer based) Program Evaluation and 
view Techniques/Critical Path Method (PERT/CPM) or 
ough a simple paper-and-pencil technique such as a Gantt 
art. A Gantt chart, for example, can be one sheet of paper 
which an activities-by-time grid is constructed to show 
1en each project activity is scheduled to start and finish, 
10 is responsible for each activity, and the key milestones 
1 ing the project. As the project progresses, actual time lines 
2 marked in. This identifies discrepancies for timely 
rrective action. 

4 

oject control is another function of particular interest to 
aluators because we usually work under strict deadlines 
id, of course, need to satisfy the client. Effective project 
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control helps ensure that the actual project results conform 
to the desired results. The project control cycle involves: 


e specifying outcome or performance standards; 
e providing continual monitoring during the project; 


e periodically comparing actual results against the standard; 
and 


e taking timely corrective action when actual results are 
failing to meet expected results. 


Techniques for exercising control include Project Status 
Reports and Milestone Reports during the life cycle. Also, 
project audits or evaluations can be conducted following a 
series of completed projects. 


From this brief discussion, it should be apparent that formal 
project management approaches should help increase the 
efficiency and effectiveness of evaluations and that many 
simpler useful tools are available to ensure the thorough plan- 
ning and management of evaluations. Interested readers may 
wish to obtain books or other materials by major contribu- 
tors to the project management literature; for example, David 
Cleland (Systems Analysis and Project Management, 1983), 
Harold Kerzner (Project Management for Executives, 1982), 
Charles Martin (Project Management: How to Make It 
Work, 1976) or The Project Management Journal. 


We, in the Review Directorate of the Public Service 
Commission, intend to conduct a project management work- 
shop in the Fall for our staff and then to evolve procedures 
and techniques to meet our particular needs. We will update 
you next year on our experiences and progress. We would 
also like to hear about your experiences with project 
management - just drop me a line. 


Robert Loo 


Robert Loo is a Program Evaluation Manager with the 
Public Service Commission. 


THE 1988 ANNUAL CONFERENCE 
THE CANADIAN EVALUATION SOCIETY 


A record 431 attended the Canadian Evaluation Society’s 
ninth annual conference in Montreal May 16-18, 1988. 
The conference - focussing on the theme “Evaluation with 
Limited Resources” - was organized in collaboration with 
the Société québécoise d’évaluation de programmes. 


The Conference Committee was chaired by André Laurion, 
Ministère de I’Enseignement supérieur et de la Science, 
Gouvernement du Québec. Conference subcommittees were 
chaired by Natalie Kishchuk and Francine Rancourt 
(Program), Kenneth Cabatoff (Workshops), Lyne Raymond 
and Nelson Potvin (Local Arrangements, Grants and 

Donations), and Jack Ivan Williams (Treasurer) in collabo- 
ration with Normand Messier. 


Promotion for this year’s Conference included CES’s first 
press conference, which was conducted in Montreal by André 
Laurion and Guy Leclerc (OCG) on May 3rd, and was 
reported in Montreal’s business newspaper ‘Finance’. 


An address by Society President Greg Mason (University of 
Manitoba) opened the Conference. Recognizing the 
outstanding progress made by the Society over the decade 
of its existence, the President stressed the need for Society 


members to view themselves as professionals rather than as’ 


a “bag of tools”. He also noted the need to understand more 
fully the political process in order to be as effective as 
possible. 


Eleanor Chelimsky (U.S. General Accounting Office) was the 
plenary speaker on the second day of the Conference. She 
emphasized the need for evaluators to understand the infor- 
mation requirements of their parliamentarians. In particu- 
lar, she noted that gathering and organizing evaluation 
information into a short, succinct, synthesis provided an 
effective vehicle to bring to the attention of legislators lessons 
learned from past experiences. In this way evaluation infor- 
mation can have a considerable impact on proposed 
programs before legislation is passed. 


Jeffrey Simpson, a columnist with The Globe and Mail, 
delivered the closing luncheon address. He observed that 
evaluation is less effective when there is no political will to 
change, and that political will is often inversely related to 
the amount of public attention focussed on the proposed 
changes. 


Those involved with evaluation at the federal level who led 
workshops were: 


* Robert Riendeau (Destrier): evaluation planning - 
resources and constraints; 


° David Zalinger (Peat Marwick): selection bias modelling: 


e Avrim Lazar and Wayne Moore (PSC): surviving the 
politics of program evaluation; 


¢ Norman Bloom and Pamela Riseborough (B and B 
Editorial): a little bit of editing goes a long way; 


+ Nancy Staisey (Price Waterhouse): implementation anal- 
ysis - evaluating the process of program building; and 


¢ Robert Bergeron (DFO) and Phil Carr (Woods Gordon): 
management of an evaluation group. 


Guy Leclere (OCG) moderated a special panel of invited 
guests, who discussed “Evaluation Development in Three 
Canadian Provinces’. The special panelists were Gilles 
Bédard, Bureau du Vérificateur général, Gouvernement du 
Québec, Mark Cox, Management Board of Cabinet, Govern- 
ment of Ontario and Zena Vigod, Department of Justice, 
Government of New Brunswick. 


e Ken Traversy (OCG): marketing program evaluation; 


¢ Michael O’Brecht (MRC): when you are the “unit”. 


Those involved with evaluation at the federal level ' 


contributed to technical sessions included: 


Hussein Rostum (Teega Research), Jennifer Stewart (S8} 
Cameron McQueen (OCG), Nancy Staisey (Price Ws | 
house), Neil Maxwell (OAG), Terry Burbridge (ED | 
Benoit Gauthier and David Black (DOC), Toru U 
(OCG), Michel Laurendeau and Sayed Shelbaya (H 
Ken Watson (Rideau Research), Frank Graves and Dou 
Hall (Ekos Research): evaluation with limited resou { 
° John Mayne (OCG), Bill Baker (RCT), David Bl} 
(DOC) and William Smith (PA): establishing and renev 
an evaluation function - the federal experience; 


ui 


e Harry Needham and Ian Kelly (SGC): the political con 
of program evaluation; 


Gary Barber (EMR) and Gary Savant (SC): data anal 


- approaches and questions; | 


| 

¢ François Lacasse (Université du Québec à Hull), sc | 
Gauthier and Shelley Borys (DOC): evaluation of hum 
resource management programs; | 

¢ Garry Sears (Peat Marwick), Ronald C. Corbeil (OC 
Carol Motuz (AGR) and Jim Asplin (RCCE 
performance measurement in the federal government | 
Filling the void; | 1 

| 

| 

| 

| 


¢ Louise Crandall (NCC): evaluation at the local level: 


e David Black (DOC), Frank Graves and Donald Hall (EK 
Research): scarce resources in the evaluation of cultu 
programs; | 


e Len Rutman (Price Waterhouse), Guy Leclerc (OC 
Joyce Potter (CMHC): evaluation perspectives on | 
CCAF’s effectiveness audit report; 


e Autar Wasson (Canada Post) and Suzanne Tascherea 
(IDRC): evaluating training in organizations, 


¢ Ike Jaswal (EMR), Donald Hall (Ekos Research) and Ste le 
Montague (Young & Wiltshire): management of eval ui 
tions with limited resources; 


¢ Barry Bragg (PSC), Len Rutman (Price NUE SR 
Gordon Cassidy (Gordon Cassidy Consulting) a 
David Black (DOC): getting the most for your a in 
dollar; 


D. 


¢ Bob Allen (HWC): evaluation of behavior modificat fe 
programs; 


EWS BRIEFS 


in Information Exchange Seminar on the Evaluation 
f Legislation was held on February 18, 1988. 
ob Wilson, Director General of the Bureau of Program 
valuation and Internal Audit at the Department of 
hstice, traced the work that had been undertaken by the 
lepartment to develop a comprehensive framework for 
aluating legislative programs, the result of which was 
esented in a draft document entitled Checklist for 
valuation of Legislative Programs. The consultant who 
repared the Checklist, Eric Milligan, then took the 
articipants (approximately 40) through the document, 
ffering guidance on how to use it. Wendy Bryans, a 
‘rogram Evaluation Manager at the Department of 
stice then illustrated how she had recently used the 
thecklist. The session was organized and animated by 
lonald C. Corbeil, Program Evaluation Branch, OCG. 


| June 1988 two officials from the Australian Government 
isited the OCG. On June 2nd, Brian Corcoran, 
Department of Finance, met with Guy Leclerc and 
lohn Mayne (PEB) to discuss matters of program evalu- 
tion. He also met with Doug Wood (Audit & Review 
Branch of the OCG) and officials of the Personnel Policy 
Branch of TBS to discuss issues related to resource 
anagement practices and management efficiency. From 
lune 6-8, Graham Carpenter, Comptroller General of the 
state of Victoria, Australia met with officials of TBS, 
finance and the OCG. They discussed rate of return 
eporting, the role of the Auditor General in value-for- 
noney and comprehensive auditing, accounting for 
financial instruments, and performance measurement. 

| 

Dn November 2, 1988, the National Capital Chapter of 
he Canadian Evaluation Society (CES-NCC) will hold 
ts Third Annual One-Day Workshop at the Ottawa 
Congress Centre. The keynote speaker will be Harry 
Rogers, Deputy Minister, Regional Industrial Expansion. 
Plan to attend! 


Do you have an article of interest on program 
evaluation that you wish to submit for publication? The 
Newsletter welcomes all contributions from the 
evaluation community. For further information, please 
call the Editor, Paulette Somerville at (613) 957-7167. 


IMINGS AND GOINGS ... 


PEB welcomes Hervé Déry and Stephen Tsang as 
Evaluation Analysts with the Operations Division. Hervé 
was formerly with Fisheries & Oceans Canada and 
Stephen came to us from Treasury Board Secretariat. 


e A farewell note to Alain Barbarie, Senior Policy Analyst, 
Program Evaluation Branch, OCG, who has recently been 
appointed Director of the Program Evaluation Group at 
Energy, Mines & Resources. Alain has just completed an 
Interchange Canada assignment with the Library of: 
Parliament. 


e PEB is wishing all the best to Lee McCormack, Senior 
Analyst of the Evaluation Operations Division, who has 
joined the Audit & Review Branch of the OCG. 


NINTH ANNUAL CONFERENCE 


CANADIAN COMPREHENSIVE AUDITING 
FOUNDATION 

The Centre Sheraton - Montreal, Quebec 
November 27-29, 1988 


Like previous CCAF annual conferences, this ninth one is 
again concerned with the issue of accountability and how 
it can be improved and strengthened. This year’s conference 
theme “Management and Auditors - A Team Approach” 
offers managers, auditors and their clients - and others, 
including members of the evaluation community - an unique 
opportunity to see how their colleagues are collaborating in 
a variety of public and private sector organizations to 
improve information on their organizations’ performance. 


The Opening Plenary session will highlight what managers 
are doing and have already done to improve the quality of 
performance information that federal government depart- 
ments are producing. As well, the session will explore the 
roles the key players in the accountability chain play in 
producing that information. The session will also provide 
a Central Agency perspective on the need for information 
on performance. 


Several of the concurrent sessions will be of particular interest 
to program evaluation specialists, including one on “The 
Senate Report on Comprehensive Auditing”, which presents 
a federal perspective on defining “accountability expecta- 
tions”. Other concurrent and special-interest sessions also 
explore performance reporting in different environments: the 
private sector, Crown corporations and provincial and 
municipal governments. For more information contact: 


Canadian Comprehensive Auditing 
Foundation 

55 Murray Street, 2nd Floor 

Ottawa, Ontario 

KIN 5M3 

Telephone: (613) 236-6713 


: SCHEDULE AND DESCRIPTION OF EVENTS 


The workshops and seminars listed below will be given in the language(s) indicated. 


WORKSHOPS AND SEMINARS 


METHODS FOR DETERMINING 
PROGRAM OUTCOMES 
(French and English) 


Purpose 


This three-day course provides evaluation practitioners with 
a review of the advantages, disadvantages and appropriate- 
ness of a range of available methods for determining program 
outcomes. In addition to attending lectures, participants 
engage in problem-solving workshops and make presenta- 
tions on the results of these exercises. A departmental 
program evaluator will present an example illustrating what 
methods and evaluation designs were used in an actual, 
completed program evaluation study and will discuss the 
issues, problems and results arising from the program 
evaluation. 


Content 


e Evaluation design 

e Data collection 

e Analysis techniques 

© Developing evaluation designs 
e Developing evaluation strategies 


e Formulating conclusions 


Workshop in French 
Date: September 27-29, 1988 


Location: Government Conference Centre, 
Committee Room, 2nd Floor, 
2, Rideau Street, 
Ottawa, Ontario. 


Workshop in English 
Date: November 22-24, 1988 


Location: Government Conference Centre, 
Colonel By Room, 3rd Floor, 
2 Rideau Street, 
Ottawa, Ontario. 


Purpose 


_® Understanding of the role of qualitative data in genel f 


*1 
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QUALITATIVE METHODS 
(English and French) 


Most evaluation studies involve the collection and anal | 
of qualitative data. This workshop is intended primarily} 
evaluation directors and senior project managers, to prom@l| 


ing evidence in evaluation studies | 
e Discipline and rigour in the use of qualitative meth@] 


e Appreciation of how qualitative and quantitative data ¢ 
be used together to advantage 


Content 


e Data sources and data collection methods: interview 
open-ended self-administered questionnaires, grow 
opinion (expert panels, focus groups), observati¢ 
document and file reviews 


e Methods of data recording and coding 
e Quantifying qualitative data 

e Analysing qualitative data 

e Credibility of evidence based on quantitative data 


e Uses and misuses of qualitative data in different types 
evaluation studies | 


e Complementarity of qualitative and quantitative dat | 


e Reporting of qualitative data 


Workshop in English 
Date: November 3-4, 1988 


Location: Government Conference Centre, 
6th Floor, Conference Room, 
2 Rideau Street, 
Ottawa, Ontario. 


Workshop in French 
Date: December 1-2, 1988 


Location: Government Conference Centre, 


Rideau Room, 5th Floor, 2547 
2 Rideau Street, # 
Ottawa, Ontario. | 


tVEY METHODS AND 

ESTIONNAIRE DESIGN 

blish and French) 

j | 

pose 

ny evaluation studies use surveys to collect data about 
ogram. The ease of conducting a survey and the useful- 
of its results are enhanced by expert design, careful plan- 
r and valid sampling. Together with Statistics Canada, 
OCG will hold a three-day workshop for those wanting 
lintroduction to surveys, questionnaire design and 
pling in an evaluation context. 


rhe elements of good questionnaire design 
yampling 

Working with consultants 

Bovernment regulations on the opinion information 
jollection | 

survey data analysis 


Ë 


iffective presentation of data 


rkshop in English 
fe: January 10-12, 1989 


ration: Government Conference Centre, 
|A Committee Room, 2nd Floor, 

2 Rideau Street, 

Ottawa, Ontario. 


rkshop in French 
te: February 7-9, 1989 


ation: Office of Human Resources, 
Room 123-C&D, ‘G” Complex, 
1039 de la Chevrotière, Ist Floor, 
Quebec City, Quebec. 


INFORMATION EXCHANGE SESSIONS 


Norm Steinberg, Director of Evaluation, Transport 
Canada, will lead a session on Evaluating Internal 
Communications, the first of the Fall/Winter series. 
This seminar will be held on Thursday, October 20, 
1988 in the East Tower, 8th Floor Boardroom, 
D planade Laurier, at 2:00 p.m. 


We anticipate holding these sessions on a monthly 
basis. Those with experience that they would like to 
share with interested members of the community are 
aoe to contact Rob Jones at 957-7229. 


PROGRAM EVALUATION 
ORIENTATION WORKSHOP 
(French and English) 


Purpose 


This five-day in-residence workshop provides an overview 
of the phases of the evaluation process. Using case studies, 
participants will have the opportunity to complete the steps 
in the evaluation process and receive feedback on their efforts 
from their colleagues and session leaders. 


All newly-appointed evaluation officers are encouraged to 
participate in this core workshop. It covers material consid- 
ered to be preparatory to other PEB seminars and 
workshops. 


Content 


e The Context for Program Evaluation 
e The Program Component Profile 
e The Evaluation Assessment 


e Generating a list of questions 
e Choosing appropriate evaluation approaches 
e Developing evaluation options 
e Writing the assessment report 


e Implications for Evaluation Frameworks 
e The Evaluation Study 


e Considerations in data collection 

e Producing findings and drawing conclusions 

e The format and content of the executive 
summary and study report 

e Implementation of results 


Workshop in French: A three-day non-residence pilot 
workshop covering the same general content as the English. 


Date: February 21-23, 1989 


Location: Government Conference Centre, 
Committee Room, 2nd Floor, 
2 Rideau Street, 
Ottawa, Ontario. 


Workshop in English 


Date: March 19-24, 1989 


Location: Donald Gordon Centre, 
421 Union Street, 
Kingston, Ontario. 


Cost: $79 per night (including room and board) for five 
nights. 


SCHEDULE OF PROFESSIONAL DEVELOPMENT EVENTS 


FISCAL YEAR 1988-89 


Workshops Language | Date 


Methods for Determining French Sept. 27-29/88 
Program Outcom S 


Qualitative Methods English Nov. 3-4/88 


Methods for Determining English Nov. 22-24/88 
Program Outcomes 


Qualitative Methods French Dec. 1-2/88 


Survey Methods & English Jan. 10-12/89 
Questionnaire Design 


Survey Methods & French Feb. 7-9/89 
Questionnaire Design 


Program Evaluation French | Feb. 21-23/89 
Orientation Workshop ; 


Program Evaluation English Mar. 19-24/89 
Orientation Workshop 
(in residence) 


TO REGISTER 


Location 


Govt. Conference ce | 
2 Rideau Street, 
Ottawa, Ontario. 


Govt. Conference Cent 
2 Rideau Street, 
Ottawa, Ontario. 


Govt. Conference Centr} 


_ 2 Rideau Street, 


Ottawa, Ontario. 


Govt. Conference Centr] 
2 Rideau Street, 
Ottawa, Ontario. 


Govt. Conference Centn 
2 Rideau Street, 
Ottawa, Ontario. 


Office of Human 
Resources, : 

Room 123 C&D 

“G” Complex 
1039 de la Chevrotière. 
Ist Floor, 

Quebec, Quebec. 


Govt. Conference Centr 
2 Rideau Street, | 
Ottawa, Ontario. 


Donald Gordon Centre, 
421 Union Street, 
Kingston, Ontario. 


Priority will be given to those who are associated with the federal program evaluation function. If you wish to register : 
for these sessions, please contact Joan Cater at (613) 957-7180. 
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Ontario Approves a 


Program Evaluation Policy 


On September 27, 1988, the Manage- 


ment Board of the Government of 


Ontario approved a policy on activity 
review, requiring ministries to evalu- 
ate their programs. The policy came 
at the end of a two-year initiative by 
the Management Board to establish 
activity review as a recognized 
management process with the Ontario 
government. The minister responsi- 
ble, Murray Elston, discussed the 
policy and the government’s inten- 
tions at the opening session of a 
one-day conference of the Toronto 
Chapter of the Canadian Evaluation 
Society on October 6. 


While some of the terms used differ 
from those used at the federal level, 
the policy is quite similar to the 
federal Treasury Board policy on 
evaluation, with the addition of some 
central capacity to evaluate. 


Activity review refers to systematic 
studies assessing the adequacy and 
relevance of program objectives; the 
effectiveness of the programs used 
to achieve objectives; and the 
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appropriateness’ and efficiency of 
mechanisms used to deliver programs 
to the beneficiaries. Activity review is 
to be done by the ministries primari- 
ly to provide information for internal 
decision-making, with the deputy 
head as the main client. Ministries are 
to assign responsibilities for coor- 
dinating and monitoring the activity 
review function, and evaluation 
requirements for new programs are to 
be part of the design for new or modi- 
fied programs. Ministries are to 
submit annual plans to the Manage- 
ment Board, as well as a report on the 
status of past evaluations and what 
program actions have in fact 
occurred. 


Mr. Elston discussed program evalua- 
tion as an ongoing responsibility of 
management and not a ‘‘fad’’ that 
will soon pass. All ministries have now 
a person responsible for activity 
review, and the staff of the Manage- 
ment Board was monitoring over 
100 reviews. Of 25 reviews complet- 
ed, 50 per cent had confirmed the 
status quo and 50 per cent had result- 
ed in either termination or substan- 
tial modification of the program. 


Another feature of the Ontario 
approach is provision for both across- 
ministry reviews coordinated by the 


Program Evaluation 


Management Board Secretariat and, at the request of 
Cabinet, for evaluations to be carried out by the same 
board. In both cases, the evaluations/reviews are carried 
out by individuals seconded from ministries to the 
Management Board for short periods of time. 


There is one more component to Ontario’s approach. 
A separate ‘base review’ process is also being established 
to provide the government with the flexibility it requires 
to fund new and high priority programs out of existing 
budgets. All ministries are to evaluate 10 per cent of their 
base annually with a view to identifying possible cost 
savings. Base review and activity review will ensure that 
evaluation has an ongoing management role in the 
Government of Ontario. = 


John Mayne 


Mr. Mayne is Director, Policy Division, 
Program Evaluation Branch, OCG 


Most Evaluation Studies 
Put to Good Use 


The results from recent OCG research into the use of 
evaluations in the federal government are very impres- 
sive. Virtually all completed studies have some impact. 
Almost three-quarters of them materially affect the 
program being reviewed. 


Program Evaluation Branch (PEB) has a continuing 
project examining the utilization of evaluation. We are 
looking mostly at the effect of the evaluation process on 
the evaluated programs, although another important 
interest is accountability to central agencies, Parliament 
and the public. In tracking the use of evaluations, PEB 


liaison.officers follow up with directors of evaluation on. 


the impact of significant studies. Brief summaries, based 
on these follow-ups, are written about the background, 
findings and results of each study. This year as well, two 
special studies are being carried out on the use of 
regulatory evaluations and S&T evaluations. 


As of February 1989, results information had been 
collected for 275 of the 543 evaluations completed 
between April 1981 and March 1988 in the 40 or so major 
departments and agencies being monitored. A further 
4] are excluded from tracing because they are classified 
or are of a specialized nature. The rest (227) require more 
follow-up. 


Instrumental Use 


The summaries are coded with a few key words to 
indicate the type of use. The categories are mutually 
exclusive. Where two terms could apply, the stronger 


category was used. For example, termination is used! 
where evaluation played a part in the decision to close | 
down a program. Reform describes a study’s association | 
with major changes in the design or scope of a program. | 
Modification indicates that more modest changes were | 
implemented to improve efficiency and/or effectiveness. | 
These three categories - termination, reform and modifi- 
cation - describe‘instrumental utilization’. Almost 74 per | 

cent of the studies that have been traced can show some | 
instrumental use or direct impact on the program. | 


: 
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Conceptual Use | 
The key word confirmation captures those situations | 
where the evaluation figured in an active decision not | 
to change the program. This includes the extension of | 
a sunsetted or pilot program. Increased understanding . 
indicates that some new information about the program ! 

was discovered and communicated which may eventu- | 
ally influence policy or program decisions. These two | 
less substantial categories are sometimes described as 
‘conceptual utilization’ and they account for another 
23 per cent of the traced studies. Note that such use may 
be extremely valuable in reorienting thinking about | 
programs or in laying the groundwork for future changes. | 


UTILIZATION SCOREBOARD 


Evaluations Tracked for Results 
1981-82 to 1987-88 


Instrumental Use 


Termination 
Reform 
Modification 


Conceptual Use 


Confirmation 
Increased Understanding 


No Results 


The above program consequences were associated with, 
but not necessarily solely attributable to, completed 
program evaluations done in major departments and 
agencies. The sample is about half of all completed evalu- 
ations but may not be representative of the whole. 


The remaining nine studies are classed as no results. This 
category is reserved for studies which produced no 
valuable information or which were never reported, 
perhaps because a program was cancelled, transferred 
or reorganized before the study was wrapped up. 


Thus, close to 97 per cent of the studies traced were 
reported as ‘utilized’ in some respect, most often in 
direct improvements. This is not to say that the maxi- 
mum utility is being extracted from every evaluation or 
that all stakeholders - program clients, program staff, 
politicians and central agencies as well as program 
management - are benefitting as much as possible. Some 
| validation of this encouraging pattern may come from 
| the more detailed research in the S&T and regulatory 
| areas and from tracing the results of the remaining 
| Studies in this time period. 


To keep PEB Newsletter readers informed of the progress 
| of this project, a ‘box score’ will be published each issue. 
An example of the utilization of an evaluation study will 
| be included as well. 


Example 


This issue's example is from Indian and Northern Affairs 
Canada. In the spring of 1988, the department complet- 
ed its evaluation of the Youth Entrepreneurship Program. 
This program selected and trained Indian youth with the 
aim of setting up small businesses. Contributions were 
provided for training in preparing business plans, for 
funding start-ups of promising businesses and for some 
| Consulting support after the businesses were set up. 


The evaluation looked at a number of pilot projects for 
this new program and then measured the results 
achieved by each (i.e., the number of new businesses 
established by individuals who completed the program 
and the cost for each new business). The best practices 
from each pilot were identified and have been used to 
redesign the entrepreneurship program. The favourable 
findings in the report were used by program manage- 
‘Ment to seek support for a continuation of the program 
(with a design modified as recommended in the evalua- 
tion). Because of the changes in design and scope for the 
program, this evaluation was classified under the reform 
category. = 


John Batchelor 


Mr. Batchelor is a Senior Policy Analyst with the 
Program Evaluation Branch, OCG 


OECD Workshop: 
Evaluation of S&T Policies 
and Programs 


On November 28 and 29, 1988, the Organization for 
Economic Cooperation and Development (OECD) held 
a workshop on the evaluation of science and tech- 
nology policies and programs. Sponsored by the OECD 
Directorate for Science, Technology and Industry, the 
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workshop involved representatives from 20 countries 
and international organizations. The undersigned were 
the Canadian delegates. 


This workshop was a follow-up to one held in late 1987 
concerning the evaluation of innovation programs. 
Dr. Barbarie reported on this earlier workshop in the 
March 1988 Newsletter. 


The stated purpose of the 1988 workshop was to extend 
and deepen understanding of the current state of S&T 
evaluation in OECD member countries. Beyond serving 
as a basis for a useful exchange of perspectives, the meet- 
ing provided delegates with an opportunity to identify 
focal points around which various actions to improve the 
state of practice might be considered. 


The agenda for the workshop was wide-ranging. A 
discussion document related to current experience in 
selected OECD countries was reviewed. The approach 
of the Canadian federal government was one of those 
highlighted in the document. Other countries discussed 
include: United Kingdom, Japan, the Nordic countries, 
Switzerland and West Germany. In its preliminary 


. conclusions, the report makes the case that the socio- 


political context of individual countries has a strong 
influence on the nature of S&T evaluation and that this 
may inhibit transference of concepts and practice. 
Nonetheless, some common focal points were identified: 


¢ evaluation is a management tool: 


* evaluation needs to be seen as more than an ‘‘ex post"’ 
control - its true value lies in its ‘‘real time’’ applica- 
tion aimed at policy and program improvement: 


¢ methodological principles to guide international 
practice are needed; and, 


¢ evaluation needs to be tailored to its particular 
environment and objectives. 


As federal government evaluators, we can immediately 
identify with these points as central to the conduct of 
program evaluation. Indeed, Canada is generally regard- 
ed as having, perhaps, the most extensive and systematic 
approach to S&T evaluation within the OECD. Our 
infrastructure and experience are considerably advanced 
in relation to most countries, and, as such, we are looked 
upon as an important source for perspective and advice. 


Another portion of the workshop's agenda was devoted 
to presentations by individual countries concerning their 
S&T evaluation practices. Mr. Moynagh presented a 
paper entitled, ‘‘Evaluation of Science and Technology 
Activities in Canada’’.* Papers related to the United 
Kingdom and Japan were also tabled. 


* Please contact the Editor of the Newsletter for a copy of this 
paper. 
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The central messages of Mr. Moynagh’s presentation were 
as follows: 


e Science and technology activity is pervasive within 
the federal departmental structure. In recognition of 
this, specific guidelines for evaluating this type of 
programming have been developed. More recently, 
S&T has been identified as a separate sector for evalu- 
ation strategy within the government and within 
OCG. 


e Significant areas of interdepartmental S&T activity 
have been identified for examination through 
program evaluation (e.g., major survey programs, 
extramural performance). 


e Over the past five years, a great number of S&T evalu- 
ation studies have been completed. The OCG is now 
engaged in a research project to examine the nature 
and utilization of this evaluation work. 


e Our intention is to revise evaluation guidelines for 
science and technology to take into account the 
results of the S&T utilization study referred to above 
and to reflect the government’s current S&T policy 
and management framework. 


There were a number of other matters discussed at the 


workshop including the approach to the ‘‘country 
reviews’’ that the OECD does of individual countries’ 
S&T and innovation policies, and other specific evalua- 
tion projects. 


A final matter discussed at the meeting concerned the 
appropriateness of beginning to describe and codify S&T 
evaluation practice within the context of guidelines or 
operating principles. A draft document was tabled to 
help focus discussion. While most delegates favoured this 
initiative, they also felt that further refinement and 
discussion on these principles would be necessary before 
their promulgation. The general direction of these guide- 
lines appears very much in accord with current think- 
ing in Canada. 


Later this year, it is likely that the OECD will publish a 


revised version of its document related to S&T evalua- 
tion in select countries. = 


David Moynagh 
Alain Barbarie 


Mr. Moynagh is Chief, Science, Economic and Regional 
Development, Operations Division, PEB 


Dr. Barbarie is Director of Program Evaluation at EMR. 


Program Evaluation and CAP 


In the fall of 1988, PEB responded to an offer by Public 
Service Commission officials to contribute to the module 
involving evaluation in the Career Orientation Program 
(CAP). A format was designed and arrangements made 
for the one-half-day event on November 23, 1988. 


Guy Leclerc, Deputy Comptroller General, opened the 
session, held in French, with an overview that empha- 
sized ways by which line managers can and shougl 
maximize benefits for themselves. | 

| 
The second part of the session dealt with practicalities 
of the evaluation process, using evaluation activity 
associated with the Salmonid Enhancement Program 
(Fisheries and Oceans Canada) as an example. Aimée 
Lefebvre-Anglin, ADM of the program at the time (now 
at Agriculture Canada), and Robert Bergeron, Director 
of Evaluation, shared their perceptions of the process 
and causes for some of the problems encountered. 
Participants learned ‘‘first-hand”” that while evaluations 
do not always proceed as smoothly as planned, many of 
the problems encountered can be eliminated or 
minimized through the active involvement of line 
managers. 


PEB thanks Mrs. Lefebvre-Anglin and Mr. Bergeron for 
their involvement. a 


OCG Practicum Sessions on IMAA 
and Program Performance 


The first two sessions of the 1988-89 OCG Practicum 
series provided (1) an update on the implementation of 
the IMAA management regime (October 28), and (2) an 
account of program performance expectations and 
measurement approaches within IMAA (December 7). 
The Practicum session on March 21 will carry forward 
the theme of reporting on program performance. 


The session in October involved presentations on 
progress made in putting the IMAA regime in place in 
particular departments and in the various branches of 
TBS/OCG by: Gérard Veilleux, Secretary of the Treasury 
Board; J. Andrew Macdonald, Comptroller General of 
Canada; Ruth Hubbard, Deputy Minister, Revenue 
Canada - Customs and Excise; and Alain Gourd, Deputy 
Minister, Department of Communications. 


The December session dealt with program accountability 
within IMAA, specifically: (1) new perspectives on OPFS 
and IMAA MOUs; and (2) ways for a department and 
TBS/OCG to define together selective and challenging 
program performance expectations and a cost-effective 
measurement and reporting strategy within IMAA, 
involving departmental program evaluation in a central 
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way. Presenters at this session were: Barry Lacombe, 
Program Branch, TBS; John Mayne, Program Evaluation 
Branch, OCG; Michael Binder, Communications Canada; 
David Moynagh, Program Evaluation Branch, OCG; Jean 
Bélanger, Communications Canada; Michael Joyce, 
Program Branch, TBS; and Jim McCarthy, Financial 
Management Information and Systems Branch, OCG. = 


_Beneñit-Cost Analysis in 


Program Evaluation 


In the mid-1970s there seemed to be a consensus that 
benefit-cost analyses (BCA) were going to become 
commonplace in resource allocation decisions of the 
government - either as an integral part of a centralized 
decision-making process (PPBS at the time, and later 
PEMS), or in response to internal department-specific 
demands. While this obviously hasn’t happened, BCA has 
been used extensively in selected areas such as major 
Crown projects and major regulatory proposals. Indeed, 
the largest single initiative that mandated BCA as an 
evaluation technique was the government’s socio- 
economic impact analysis (SEIA) policy, which from 
1978 to 1986 required the production and publication of 
SEIAs for major social regulations prior to their 
promulgation. 


Program Evaluation 


regulatory review and decision-making, and to assess the 
usefulness of these techniques for different countries’ 
needs. The discussion centred on both the scrutiny of 
proposed regulations prior to their implementation, and 
the review of the existing stock of regulation. Consider- 
able interest was expressed in Canadian procedures for 
the latter, as we have had the most experience. with 
centralized evaluation and review in this area. 


Finally, in November, the annual one-day workshop 
of the National Capital Chapter of the Canadian 
Evaluation Society included a panel discussion entitled 
‘The Scope for Benefit-Cost Analysis in Program 
Evaluation’. The panel was chaired by Fred O’Riordan, 
and members included: David Sewell, Department of 
Finance, Ian Midgley, Employment and Immigration, and 
Bruce McKellips, Treasury Board Secretariat. Topics 
discussed included: the importance of quantification in 
evaluation; the role of economics and economists in 
benefit-cost analysis; and, the reasons why BCA isn’t 
more widely used and what can be done about it. One 
area of consensus among panelists was the value of BCA 
in identifying and comparing program alternatives in an 
objective and comprehensive manner. # 


CES 1989 
Annual Conference 


Apart from these rather specialized applications, 
however, relatively few program evaluations or ex-post 
reviews have utilized benefit-cost methodology. PEB 
Officers recently participated in several events focuss- 
ing on BCA, in order to help promote a better under- 
| Standing of its potential applications and its limitations. 
In June 1988, Hervé Déry and Fred O’Riordan present- 
ed a workshop on the use of benefit-cost analysis in 
 regulatory programs as part of a two-day seminar 
Sponsored by the Office of Privatization and Regulatory 
Affairs. The workshop drew attention to the results of 
two studies: the first, a study done for Fisheries and 
Oceans to estimate the impact of acidification on the 
economic value of recreational fishing in Québec; and 
the second, done by Agriculture Canada, a BCA of 
alternative disease-control programs to eliminate brucel- 
losis in cattle. 


The Annual CES Conference will be held in 
Fredericton, New Brunswick, on May 29-31, 
1989. The theme will be ‘‘Evaluation: Towards 
Accountability’’. The Conference will focus on 
the demands for accountability which face 
individuals, organizations and governments 
today. It will concentrate on an examination 
of the nature of these demands and effective 
Strategies for dealing with them. 


For more information you may contact: 


In October, Fred O’Riordan was a member of the host 
Canadian delegation in Montréal at an OECD-sponsored 
workshop on Regulatory Analysis Techniques. This 
meeting was hosted by the Office of Privatization and 
Regulatory Affairs in association with the OECD's 
Technical Cooperation Committee, and was attended by 
representatives of nine member countries, the OECD 
Secretariat and the European Economic Commission. 
The objective of the three-day workshop was to explore 
applications of advanced analytical techniques for 


Harriet Lenard, 

Policy Secretariat, 

Government of New Brunswick, 
P.O. Box 6000, 

Fredericton, N.B. 

E3B 5H1 

Tel.: (506) 453-2071 


oO 
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Evaluating Internal Communications 


Transport Canada (TC) has undergone a period of major 
change as a result of economic deregulation and as a 
result of the expenditure reduction exercise. In light of 
these changes, a thorough review of the department's 
internal communications was undertaken, as open and 
clear communication is vital to the future strategic direc- 
tion of the department. 


In an Information Exchange Session on October 28, 1988, 
Norm Steinberg, the Director of Evaluation at TC, and 
Nick Gwyn, of J.F Hickling, presented the methods used 
in the department’s review, the main results, and some 
of the ensuing actions undertaken. Of particular interest 
were the nature of the issues selected to be addressed 
and the methodologies used. The issues were as follows: 


1. Is internal communications perceived as a serious 
problem? 


2. To what extent is the problem an organizational 
as opposed to a communications problem? 


3. How does information flow vertically and 
horizontally? 


4. What kind of information do managers want to 
receive? 


5. Do managers believe the messages they are receiv- 
ing and are the messages accurate? 


6. Are messages perceived as consistent, complete and 
timely? 


7. How do managers prefer to receive this information? 


8. Are managers aware of and do they understand the 
necessity or rationale for change and reorganization? 


9. Do Transport Canada managers and staff regard 
themselves and feel part of the ‘‘corporate family’’, 
the organization, Transport Canada, etc.? 


10. How effective are various sources of information? 


Data were gathered from a questionnaire sent to all 
managers (SM-2 equivalent to ADM level) and from 
personal interviews with 150 managers across Canada. 
The data were analyzed and presented by the consul- 
tants in an “‘interim’’ report that was sent to all 
managers. The interim report was discussed at a series 
of ‘‘Town Hall’’ meetings of groups of managers across 
the country. This led to a final report on strategic objec- 
tives and specific goals, which was later presented to the 
Deputy Minister. 


The final report was used as a basis of discussion for 
identifying problems of internal communications across 
the federal government by a council of senior deputy 
ministers. The contents of the report are also being 


| 
discussed as part of senior management courses at the 


Executive Training Centre in Touraine, Québec. 


Further information with respect to this study can be 
obtained from Norm Steinberg at Transport Canada. @ 


Our readers will have noticed the Newsletter's 
new look. It reflects the recently developed | 
corporate image that Treasury Board will be | 
using for all of its publications. The company | 
Triad Design Group of Ottawa developed the 

new corporate image. 


News Briefs 


e Last November, Mr. Tim Bannatyne, Director of 
Planning from the New Zealand Department of 
Justice, visited Ottawa. During his stay he met 
with representatives of the OCG Branches to discuss 

issues in the policy and management areas. John 

Mayne, Director, Policy Division, PEB, met with 

Mr. Bannatyne to discuss the many facets of program 

evaluation. | 


e A Conference on Tax Expenditures and Government 
Policy was held at the John Deutsch Institute of 
Queen's University on November 17 and 18, 1988. 
Fred O’Riordan (PEB) was a participant at the 
Conference, which was sponsored by the Department 
of Finance and organized by David Sewell, Director 
of the Tax Measures Evaluation Division. It drew on 
papers from invited speakers from governments and 
universities in Canada and the United States. Topics 
included problems of definition and measurement 
inherent in the tax expenditure concept, experience 
to date in accounting for tax expenditures in the 
Canadian and U.S. governments, distributional 
impacts and the political economy of tax 
expenditures, and alternatives for accountability. The 
proceedings of the Conference will be published by 
the Institute (target date for availability is 
March 1989). 


Comings and goings . . . 


A farewell note to Karen Traversy, Evaluation Analyst, 
Operations Division, who has recently accepted a 
position with Fisheries and Oceans Canada. 


PEB welcomes Paul Kishchuk as Research Analyst with 
the Operations Division. Paul originates from Saskatoon 
and is presently continuing his studies in economics at 
Carleton University. 
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Professional Development Events 1989-90 


Program Evaluation Orientation Workshop (English) 


Purpose 


This five-day in-residence workshop provides an 
__ overview of the phases of the evaluation process. Using 
_ case studies, participants will have the opportunity to 
complete the steps in the evaluation process and receive 
feedback on their efforts from their colleagues and 
_ session leaders. 


All newly appointed evaluation officers are encouraged 
| to participate in this workshop. It covers material consi- 
dered to be preparatory to other PEB seminars and 
workshops. 


Content 


+ The Context for Program Evaluation 
+ The Program Component Profile 
+ The Evaluation Assessment 
° Generating a list of questions 
¢ Choosing appropriate evaluation approaches 
+ Developing evaluation options 
e Writing the assessment report 


¢ Implications for Evaluation Frameworks 


¢ Th 
& 
e 
@ 


e Evaluation Study 

Considerations in data collection 

Producing findings and drawing conclusions 
The format and content of the executive summary 
and study report 

Implementation of results 


Date: October 22-27, 1989 


Location: Donald Gordon Centre, 


Cost: 


421 Union Street, 
Kingston, Ontario. 


$79 per night (including room and board) 
for five nights 


HU gt gi aa RE ae ee 


We will be offering additional workshops in 1989-90 - to be announced at a later date. 


Do you have an interesting article on program evaluation that we can publish? The Newsletter welcomes 
all contributions in either official language from the evaluation community. For further information, please 
call the Editor, Paulette Somerville at (613) 957-7167. 
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Standards for Federal 
Program Evaluation 


Working Standards for the Evalu- 
ation of Programs in Federal Depart- 
ments and Agencies are being issued 
by the Program Evaluation Branch of 
the Office of the Comptroller General. 
This underlines the maturity and 
growing professionalism of federal 
evaluation practice. It also demon- 
strates the role that evaluation is 
expected to play in strategic manage- 
ment and accountability. This article 
outlines how the standards were 
developed, briefly describes the 
standards themselves, and discusses 
their use. 


Scope and Focus of 
the Standards 


For some time there has been a need 
for an up-to-date articulation of 
expectations for program evaluation 
which takes account of the many 
changes in government management 
processes over the past decade, 
including the increased focus on 
accountability for program perform- 
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ance. The evaluation standards both 
provide this and, at the same time, 
reflect the basic principles of govern- 
ment policy on program evaluation. 
This year, the, Office sof the Comp- 
troller General (OCG) will.be conduct- 
ing a review of evaluation policy, 
using the’ tandar indards 45 a framework. 
The review wi ed on the more 
detailed provisions of evaluation 
policy, such as the three-to-five year 
evaluation coverage of all programs 
on a cyclic basis. 


The essence of government evalua- 
tion policy is that departments and 
agencies will (a) evaluate the perform- 
ance of their program, taking into 
account both departmental and 
government concerns; (b) confirm or 
modify their programs, in the light of 
these evaluations; and (c) report 
evaluation findings and conclusions in 
a clear and balanced manner. 


The standards flesh out these 
fundamentals. They apply both to 
planning and conducting evaluation 
studies and to using evaluation 
findings. They are intended as 
reference points for evaluators and 
senior managers in managing the 
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departmental evaluation process (Departmental 
Standards), and planning, implementing, and using 
individual evaluations (Standards for Individual Program 
Evaluations). 


The first set of standards deals with the quality of evalu- 
ation practice by departments and agencies over time, 
while the second deals with the quality of the processes, 
findings and program consequences associated with 
individual program evaluations. The standards are stated 
at a high level of generality and are amplified, in most 
cases, by more specific standards. The standards address 
only implicitly the quality of professional evaluation 
practitioners. 


Meeting these standards involves the support of the 
deputy head, the organization’s senior and other 
managers, in addition to the departmental evaluation 
manager. The evaluation manager provides professional 
staff support for evaluation, particularly for: (a) plan- 
ning evaluations; (b) gathering relevant evidence on 
program performance; (c) documenting evaluation 
findings and conclusions; and (d) making recom- 
mendations. 


The organization’s program evaluation manager normally 
influences the shape and outcome of other activities 
mentioned in the standards, but may not have the 
lead responsibility for them, e.g., development of the 
Operational Planning Framework (OPF) and of an IMAA 
Memorandum of Understanding (MOU), ongoing program 
performance measurement, program decision-making 
and follow-up, and reporting outside the department on 
evaluation findings. 


Departmental Standards 


The basic departmental standards are: 


Al Measurement and Analysis 


Based on a strategic plan, the department 
credibly evaluates the performance of its. 
programs, taking into account the objectives — 
and priority concerns of both the department 
and the ever 


A2 Periodic Program Performance Review 


Departmental programs are confirmed or | 
modified in the light of program evaluation 
findings, conclusions, and FCO COA ONS 


A3 External Reporting 


The department reports program evaluation 
findings and conclusions in a clear and 
balanced way and in a manner consistent 
with Cabinet and Treasury Board submission 
procedures and external reporting 
requirements. 


ments ine departmental 
: OverRenG 


Ba Rare Reports . 


Evaluation reports present the fi 


1. ee by the a a Dent. 


The department uses the ev. uatio 
ings, conclusions, and recom 

- improve, modify, or confirm . 
evaluated and to meet any accot 
reporting requirements =  _ 


Use of the Standards 


The standards provide a basis for internal self-assessment 
and quality improvement activities by departments and 
agencies. They are also being used by the OCG in assess 
ments of individual evaluations and of evaluation gener: 
ally in departments. Such assessments, like evaluatior 
itself, are not checklist exercises, but involve disciplinec 
professional judgments. 


Development of the Federal 
Evaluation Standards 


The standards for evaluation have grown out of the rick 
experience of the federal evaluation community over the 
past decade. This experience has been nurtured by the 
OCG Guide and Principles, evaluation monographs ant 
professional journals, including The Canadian Journa 
of Program Evaluation, and several ‘‘first generation’ 
evaluation standards documents described in th 
Appendix. 


In 1986, the OCG tested an early version of the presen 
standards in connection with (a) the development of the 
first annual Government Program Evaluation Plan 


(b) assessments of regulatory evaluations; and (c) the 
Auditor General’s government-wide audit of the quality 
of evaluations. The standards are the product of exten- 
sive consultations with departments, central agencies, 
the Office of the Auditor General and others. They may 

volve as experience is gained in their use and as needs 
for program review information change. 


onclusion 


Professionalism has become one of the hallmarks of 
federal program evaluation. The values and the traditions 
f the federal Public Service and of a number of intellec- 
tual disciplines have supported the development of 
professionalism in evaluation. The strength of these 
traditions has also contributed much to the development 
of the federal evaluation standards. The creative use of 
he standards will, in turn, help enable federal evalua- 
tion to respond to the challenges of the present and the 
future. 


Appendix 


1978 Cook, T.D. and Gruder, L., Evaluation Quarterly, 
: 2(1), 5-52, ‘‘Metaevaluation Research’. 


The classic treatment of strategies for quality 
improvement throughout the evaluation process. 
This article predates the formal articulation of 
evaluation standards. 


1981 Auditor General of Canada, Auditing of Proce- 
dures for Effectiveness. 


: These audit criteria, revised in 1986 as Auditing 
| of Effectiveness Measurement, Reporting, and Use, 
are broadly comparable to the federal evaluation 
standards. Their scope is both wider than program 
evaluation - encompassing, for example, the full 
range of ongoing effectiveness measurement and 
reporting on effectiveness in Part III of the 
Estimates and narrower in that the assessment of 
‘effectiveness’ does not encompass all evaluation 
issues. 


1981 Comptroller General of the United States, Govern- 
ment Auditing Standards, United States General 
Accounting Office (GAO). 


GAO ‘‘program audits’’ are conducted for the 
Congress; they are conceptually similar to 
Canadian federal evaluations, The GAO standards, 
which were revised in 1988, overlap considerably 
with the Canadian federal evaluation standards, 
but put more emphasis on the qualifications and 
independence of evaluators. 


1981 The Joint Committee on Standards for Educa- 
tional Evaluation, Standards for Evaluations of 
Educational Programs, Projects, and Materials, 
McGraw-Hill, New York. 


These standards have influenced the practice of 
evaluation generally in the United States and else- 
where. They - and 30 detailed standards, each 
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with a useful case illustration - address the 
“utility, feasability, propriety, and accuracy’’ of 
evaluations. They are intended to apply to any 
political and administrative context, in contrast 
to the present standards. 


1981 Treasury Board of Canada, Guide on the Program 
Evaluation Function. 


1981 Treasury Board of Canada, Principles for the 
Evaluation of Programs by Federal Departments 
and Agencies. 


1982 Rossi, P.H. (ed), ‘‘Standards for Evaluation 
Practice”, New Directions for Program Evalua- 
tion, 15, Jossey-Boss, San Francisco (Standards of 
the American Evaluation Association, formerly 
the Evaluation Research Society). 


These standards have also influenced the practice 
of evaluation, broadly defined, in the United 
States and elsewhere. They - and fifty-five 
detailed standards - are organized around six 
stages in the evaluation process. They are inten- 
ded to apply in any political or administrative 
context. In contrast to the Joint Committee stan- 
dards, they emphasize procedures involving the 
assessment of program impacts and the use of 
quantitative data in evaluations. # 


Cameron McQueen 


Mr. McQueen is a Senior Policy Analyst with the 
Program Evaluation Branch, OCG. 


More Utilization Studies 


The OCG continues to look at what use is being made 
of evaluations. This is a difficult research area, and there 
is a large literature, mostly from the U.S., that discusses 
the problems in dealing with utilization. The current 
debate between Carol Weiss and Michael Patton in 
recent issues of the American Evaluation Association’s 
Evaluation Practice illustrates the different views held 
on the extent and nature of the use being made of evalu- 
ations in the U.S. 


As reported in the previous PE Newsletter, we have a 
number of studies underway, each exploring how the 
utilization of evaluations can be described and classified. 
We plan to report on the study of utilization in S&T evalu- 
ations in the next Newsletter and, later in the year, on 
the use of regulatory evaluations. Although we presented 
one classification scheme last issue, (repeated below), we 
also expect other approaches will emerge as we learn 
better how to measure and report on utilization. This 
year’s Government Program Evaluation Plan, for 
example, likely will report on utilization from a slightly 
different perspective based on a five-year review of 
utilization. 
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Bvaluagen: Documented 
84.04 to 88. 03" 


; ineeramental: Use 


Termination — 

Reform . 

Modification 
FADEeTAn _ 


: Conceptual Use 
Increased Understanding 


No Results 


The above program consequences were associated 
with, but not necessarily solely attributable t 
completed program evaluations done i in 17 maior 
departments and agencies. 


Termination is used wher: evaluation n payed a a 
part in the decision to close down a program. 


Reform describes a study: s assocation with major _. 
changes in the design or scope of a D 


Modification indicates that more modes. Gan 
were implemented to prove sie and/or 
effectiveness. — 


Confira captures those oi where the 
evaluation figured in an active decision not to 
change the program. This includes the extension 
of a sunsetted or pue program. - 


Increased understanding incientes fae some n 
information about the program was discovered : d 
communicated which may eventually. influenc 
policy or programe decisions . 


No pene this citer E is enced or udi e 
which produced no valuable information or wi 
were never reported, perhaps" becaus 
program was cancelled, transferred or reorganiz 
before the study was wrapped Up. 


One aspect that will distinguish this work from much of 
the U.S. research is the emphasis on measuring and 
documenting what is called instrumental use. Much U.S. 
literature has abandoned demonstrating direct use of 
evaluation in changing specific programs, focusing more 
on the often longer-term conceptual use of evaluation 
in changing understanding about programs or program 


areas. And, perhaps, this is the best to be expected wher 
evaluations are carried out using external evaluatior 
resources in a diffuse, public and competitive policy- 
making process. Nevertheless, and without discounting 
the importance of such indirect use of evaluation, one 
would expect that internal evaluation units working for 
large organizations such as the Canadian federal govern- 
ment would be required to demonstrate direct 
instrumental use of their evaluations to account for their 
own performance. 


The statistics presented in the March PE Newsletter were 
positive regarding instrumental use. However, as was 
noted, they might have reflected some selection and self- 
reporting bias. Studies whose outcomes were unknown 
might well have been the ones that were not included 
in the sample. Evaluators, perhaps, were exaggerating 
their contributions or putting them in a favourable light. 
The statistics were an amalgamation of several classifi- 
cations done by different researchers, possibly leading 
to inconsistencies in categorization. 


The accompanying mini-scoreboard provides early statis- 
tics from a more structured approach to the review and 
classification of utilization which provides us with more 
solid documentation. So far, 101 evaluations have been 
reassessed. The pattern is roughly the same as our previ- 
ous results. It remains clear that evaluations are being 
used in important ways. @ 


Program Evaluation Orientation ea 
Goes to Malaysia 


In late November 1987, two senior Government a 
Malaysia officials visited Canada to participate in the 
Office of the Comptroller General’s Program Evaluation 
Orientation Workshop and to determine whether the 
Workshop should be conducted in Malaysia as part of a 
project sponsored by the Canadian International 
Development Agency. Based on their experience, they 
recommended that the Workshop be modified to suit 
Malaysian needs and be conducted in Malaysia in 
February 1989. | 
To modify the Workshop, Gordon Cassidy of Queen's 
University, the Workshop leader, visited Malaysia 
in August 1988 to collect appropriate information 
to undertake the necessary changes. The Workshop 
was conducted at the National Institute of Public 
Administration facility at Kuala Lumpur from 
19 to 24 February, 1989. The Institute has six campuse 
throughout Malaysia training some 20,000 public 
servants every year. Dr. Cassidy was assisted in conduct* 
ing the Workshop by two syndicate animators, 
Ronald C. Corbeil of the Office of the Comptrolle 
General of Canada and Edward Tunis of the Bureau of 
Management Consulting. 


Twenty senior officers of the Government of Malaysia 
who have an involvement or interest in evaluation of 
training programs participated in the Workshop. They 
came from seven different organizations: the Economic 
Planning Unit of the Prime Minister’s Office, the Ministry 
of Education, the Ministry of Labour, the Manpower 
Department, the Public Services Department, the 
Council of Trust for Indigenous People and the National 
Institute of Public Administration. 


Participants praised the Workshop giving it one of the 
highest overall ratings using the National Institute of 
Public Administration’s standard rating form. All 
respondents to the assessment felt they had benefitted 
from the Workshop and would recommend it to others. 


In discussions following the Workshop it was quite clear 
that the Government of Malaysia was pleased with it and 
will consider repeating it. Consideration is also to be 
given to translating the Workshop into Bahasa/Malaysia, 
the native language, so that more Malaysians may 
participate in it. The Government of Malaysia also asked 
for further information about program evaluation, 
particularly as it relates to policy development, and the 
area of human resource development. 


The Workshop, entitled Program Evaluation: An 
Orientation Workshop, was developed by the Office of 
the Comptroller General and was first conducted in 1982. 
Since then, it has undergone considerable development 
and has been carried out 20 times for some 
400 participants from over 100 federal and provincial 
government departments and agencies. A combination 
of lectures, case studies and syndicate exercises provide 
participants with a detailed introduction to planning a 
program evaluation and to drawing conclusions and 
recommendations from actual evaluation data. The 
participants present the results of their syndicate 
exercises to one or more senior public servants through 
two short, focussed sessions: one on planning an 
evaluation and one on drawing conclusions and 
recommendations. The next session will be carried out 
October 22-27, 1989, in Kingston, Ontario. # 


Do you have an incresdng ide. on program . 
evaluation that we can publish? The Newsletter — 
welcomes all contributions in either official 


4 language from the evaluation community. For. 
further information, please call the 
_ Paulette Somerville at (613) 957-7 167: - 
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> a good nn, | 
e be oe at 


uld we have a role to play : in the 
div process? 


Is th co a career “path for program 


evaluators? 


: Reaitmion. a will be mailed to CES — 


members in the coming months. More informa- 
tion is available from Iris Bradley, Workshop 


Chair, Labour Canada, (819) 953-0004. 
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OPFs, MOUs and Evaluation Assessments 


As part of the IMAA initiative, Operational Plan Frame- 
works (OPFs) and/or Memoranda of Understanding 
(MOUs) are to include an identification of expectations 
regarding program performance. Performance expecta- 
tions are either specific output or productivity targets 
to be met or performance issues on which evidence is 
to be provided. In the latter case, it is frequently program 
evaluations that will provide the required evidence. Thus 
a well developed OPF would include an identification of 
key evaluation issues that are to be addressed during the 
life of the MOU. Where, one might ask, does this leave 
the evaluation assessment process, during which 
evaluation issues are to be identified? 


There are two aspects to consider. On the one hand, the 
OPF would only identify those key evaluation issues that 
are of particular concern to Treasury Board. In most 
cases, the evaluation assessment process taking place at 
the time of the evaluation would identify other issues 
that are of importance to the department, and would 
confirm the importance of the OPF-identified issues. 


More importantly, the level of detail specified in an OPF- 
identified evaluation issue will normally not be enough 
to begin data collection. In most cases, there will need 
to be further clarification and detail in order to express 
a performance concern as a specific performance issue. 
Indeed, this is why it is possible to identify an evalua- 
tion issue well in advance of the evaluation (not unlike 
the case for evaluation frameworks): it is often clear if 
the main issue is the efficiency of delivery of the program 
or the extent to which its objectives are being met. 


This pre-specification of issues can assist evaluation in 
a number of respects. It provides central agency perspec- 
tives on key performance issues up front. It situates 
evaluation as a key support to the Deputy Minister’s 
external accountability. It provides a basis for considering 
early on what data and analysis need to be done to 
adequately address the issue, leading to identification 
of opportunities to improve data, benchmarks, etc. in the 
interim. Having these central issues identified going into 
the evaluation assessment should serve to make the plan- 
ning process more efficient and timely. 


A good illustration can be found by looking at a section 
of the recently completed MOU for the department of 
Communications (see below). Other parts are similar. The 
list of ‘evaluation issues’’ identified in the sub-activity 
are most likely not the only issues that will be addressed 
in the evaluation, and the level of detail given is ces 
not sufficient to ‘begin the evaluation. 


Thus the OPF can identify the main Treasury Board 
program performance concerns, while the evaluation 
assessment — as has always been the case - develops the 
detailed terms of reference for the evaluation study. = 


John Mayne 


Mr. Mayne is Director, Policy Division, UN 
Evaluation Branch, OCG. 


Department of Communications 
Memorandum of Understanding 
Research and Development Sub-Activity 


Performance Indicator 
Expenditures ($ and PYs) by core 
substantive area 


Extramural contracts, grants and 
contributions as percentage of total research 
related activity 


Significant findings or applications from 
research efforts 


Quality of research and nature and incidence 
of its use 


Continuing relevance of the R&D 


Relative costs and benefits of intramural 
R&D 


Performance Target 


Actual expenditures and trends over time 
in $ and PYs 


Current level (43%) to be maintained if 
overall funding remains stable 


Descriptive 


Quality comparable to that obtained in 
similar public sector labs 


Findings from the 1988-89 evaluation of 
research labs. Targets to be developed based 
on this evaluation. 


In the opinion of current and prospective 
clients and DOC research management, and 
in light of research conducted elsewhere. 


Findings from the 1988-89 evaluation. Targets 
to be developed based on this. 


Findings from the 1988-89 research evalua- 
tion and 1988-89 organizational study. Targets 
to be developed based on these studies. 


pid ELA 


Reporting Requirement 


Annual Management Report « 


À 
Annual Management Report 


Annual Management Report 


Evaluation findings to be 
summarized in 1990 AMR. 


Evaluation findings to be 
summarized in 1990 AMR. 


Evaluation and study 
findings to be summarized in 
1990 AMR. 


| 
| 
| 
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What's New! 


There's a new seminar among the professional develop- 
ment workshops and seminars offered by the PEB: 
Introduction à l'évaluation des programmes. The 

Seminar was given February 19 to 22, 1989 at Mont Ste- 
Marie. It is the product of the constant desire by the PEB 
to develop training tools adapted to specific needs. It was 
developed for a francophone audience and is intended 
to be distinct in themes and approach from the ‘‘Program 
Evaluation Orientation Workshop’. 


We were quick to call upon experienced evaluators and 
our Own experts in the PEB to help with the actual 
presentation of the three-day seminar. The following 
individuals, in order of appearance, answered the call: 
Guy Leclerc, from OCG, began with a discussion of 
program evaluation in Canadian government (environ- 
ment and context); Cameron McQueen, also from the 
OCG, then spoke about the ins and outs of program 
evaluation in the federal government; Alan Winberg, 
from INAC, dealt with the roles played by evaluation 
af pr a department; Alain Desfossés, from DOC, spoke 


f program evaluation impact using the Nielsen Task 
orce as an example; Benoit Gauthier, also from DOC, 
directed the workshop on evaluation issues, while 
Robert Bergeron, from F&O, led the workshop on 
evaluation strategies; Claude Létourneau, from BIC, 
conducted the workshop on communication and dissemi- 
nation of evaluation findings. 


Program Evaluation 


To conclude the workshop, but also open the window on 
new perspectives, Jacques Bourgault, a professor with 
the University of Québec in Montréal, gave a presenta- 
tion on current challenges in the evaluation field. 


The intensive three-day workshop was held in Mont Ste- 
Marie, a delightful location which many complained they 
were unable to fully appreciate. Although from differ- 
ent backgrounds, most of the 23 participants belonged 
to federal agencies rather than departments and were 
relatively new to the evaluation field. In general, the 
group left not just satisfied with the workshop but 
slightly overwhelmed by the good food provided and the 
intensity of the workshop’s content and schedule. We can 
be proud of the event’s success while promising to make 
some normal and necessary adjustments to this new 
workshop. @ 


Ibolya Galambos 


Mrs. Galambos is a research analyst with the Program 
Evaluation Branch. 
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Agriculture Canada 
Carol Motuz 
A/Director, Program 
Evaluation Division 


Atlantic Canada 
Opportunities Agency 
Keith Draper 
Vice-President, Corporate 
Services 


Atomic Energy Control Board 
Leonard Trudel 

Chief, Planning and 
Coordination 


Canada Council 

Peter Brown 

Senior Assistant Director 
and Treasurer 


Canada Mortgage and 
Housing Corporation 

Philip Brown 

Director, Program Evaluation 
Division 


Canadian Centre for 
Occupational Health and Safety 
Kash Manchuk 

Director, Finance, Publishing 
and Administration Services 


Canadian Broadcasting 
Corporation 

Yvon Gauthier 

Director, Coordination & 
Evaluation 


Canadian Centre for 
Management Development 

Alina Doucet 

Evaluation Manager 


Canadian Commercial 
Corporation 

F. Owen Kelly 
Comptroller 


Canadian Human Rights 
Commission 

Debra Young 

Director, Research and Policy 
Branch 


Canadian International 
Development Agency 

Jean Quesnel 

Director, Program Evaluation 
Division 


Canadian Radio-television 
and Telecommunications 
Commission 

William Weizenbach 

Director, Corporate Audit 
and Evaluation 


Canadian Security Intelligence 
Service 

Joanne Stoody 

Chief, Program Review 


Commissioner of Official 
Languages 

Yves Breton 

Advisor 


Consumer and Corporate 
Affairs Canada 

Bruno Bonneville 
A/Assistant Director 
Program Evaluation 


Correctional Service Canada 
Doug Borrowman 


A/Director, Evaluation and 
Research 


Department of Communications 
Benoit Gauthier 

A/Director, Program 
Evaluation Branch 


Department of Finance 
David Sewell 

Director, Tax Measures 
Evaluation Division 


Emergency Preparedness Canada 
John Pritchard 


Director, Evaluation and 
Analysis 


Employment and Immigration 
Canada 

Ian Midgley 

Director General, Program 
Evaluation Branch 


Energy, Mines and Resources 
Canada 


Alain Barbarie 
Director, Program Evaluation 
Group 


Environment Canada 
Pat Gore 


Manager, Program Evaluation 


Export Development 
Corporation 

Terry Burbridge 

General Manager, Policy, 
Planning & Evaluation 


994-1991 


1-506-857-6854 


995-5910 


598-4311 


748-4659 


1-416-572-2981 


738-6609 


992-8061 


996-0116 


990-2875 


997-0148 


997-0108 


782-0336 


995-0815 


953-6735 


995-9000 


990-4735 


992-4309 


995-2312 


994-2406 


996-2464 


994-5208 


598-2715 


External Affairs Canada 

Bill Van Staalduinen 

Director, Evaluation & 
Resource Review Division 


Farm Credit Corporation 
Paul Finn 
Head, Program Evaluation 


Federal Business Development 
Bank 

George W. Kyte 

Vice-President, Internal Audit 
& Program Evaluation 


Fisheries and Oceans 

Robert Bergeron 

Director, Program Evaluation 
Branch 


Forestry Canada 

Ron Turriff 

A/Director, Audit and 
Evaluation 


Health and Welfare Canada 
Bud Long 
Director, Program Evaluation 


Indian and Northern Affairs 
Canada 

Alan Winberg 

Director, Executive Support 
Services 

Evaluation Directorate 


Industry, Science and 
Technology Canada 

John Butler 

Director, Program Evaluation 
Branch 


Investment Canada 
C. Edward Hitsman 
Director, Corporate Services 


International Development 
Research Centre 

Douglas Daniels 

Director, Office of Planning 
& Evaluation 


Justice Canada 
Catherine Bragg 
Director, Program Evaluation 


Labour Canada 

Denise Roberge 

Director, Corporate Services 
Branch 


Medical Research Council 
Michael O’Brecht 
Program Evaluation 


National Capital Commission 
Luc Landriault 
Director, Management Audit 
& Evaluation 


National Defence 

Paula Skippon 

Director General, Program 
Evaluation 


National Defence (CSE) 

Donald Drew 

Director, Evaluation & Audit 
Services 


National Energy Board 

Claire Scott 

Director, Plans and Evaluation 
Branch 


National Film Board 

Claude Parent 

Planning & Program Evaluation 
Coordinator 

Commissioner’s Office 


National Museums Canada 
Wally Kozar 

Assistant Secretary General, 
Corporate Services 


National Parole Board 

Sheila Faure 

Manager, Evaluation and 
Statistics 


National Research Council 
William Smith 
Head, Program Evaluation 


National Search and Rescue 
Program 

David Walters 

Director, Audit and Evaluation 


National Transportation Agency 

Mike Andrecheck 

Director, Financial Services 
Directorate 


Natural Sciences and Engineering 
Research Council 

Jocelyn Lillycrop 

Manager, Program Evaluation 


Office of the Auditor General 

Ted F. McNamara 

Assistant Auditor General, 

Program Evaluation and 
Internal Audit 


DEPARTMENTAL CONTACTS FOR PROGRAM EVALUATION 


993-5829 


996-6606 


1-514-283-7300 


993-1937 


997-1107 


957-3889 


994-4311 


954-9633 


995-3452 


236-6163 


993-5003 


994-0728 


954-1817 


239-5225 


996-4251 


991-7102 


990-6845 


1-514-283-9251 


954-4250 


995-1308 


993-4758 


996-1696 


997-1520 


995-5911 


Office of the Superintendent of 
Financial Institutions Canada 
Robert Emond 

Executive Director 
Management Services Sector 


Public Archives 
Canada/National Library 

Paul Prieur 

A/Director, Program 
Evaluation and Research 


Public Service Commission 
of Canada 

Barry Bragg 

A/Director of Program 
Evaluation, Review Directorate 


Public Service Staff 
Relations Board 

Louis Labrecque 

Director of Administration 


Public Works Canada 
Don Brennan 
Director, Program Evaluation 


Revenue Canada, Customs 
and Excise 

Nicole Mendenhall 

Director, Program Evaluation 
Branch 


Revenue Canada, Taxation 

William Baker 

Director, Program Evaluation 
Branch 


Royal Canadian Mounted 
Police 

William Maffet 

Senior Evaluation Officer 

Program Evaluation Branch 


Secretary of State 
Kerry Johnston 
Director, Program Evaluation 


Social Sciences and Humanities 
Research Council 

Robert Hanson 

Coordinator, Program 
Evaluation Branch 


Solicitor General of Canada 
Harry Needham  ~ 
Director, Management Review 
Group 


Statistics Canada 

Martin Hubbard 

Director, Program Evaluation 
Division 


Supply and Services Canada 

Marie-Josée Mercier-Savoie 

Director General, Policy 
Development & Analysis 
Directorate 


Telefilm Canada 
Jean-Paul Paré 

Director, Finance and 
Administration 


Transport Canada 
Norm Steinberg 
Director, Program Evaluation 


Treasury Board of Canada 
Office of the Secretary of 
the Treasury Board 
Doug Redden 
Senior Analyst, Corporate Policy 


Personnel Policy Branch 

André Robert 

Director, Evaluation and 
Review Group 


Administrative Policy Branch 
Bill Douglas 

Group Chief, Planning and 
Evaluation Division 


Official Languages Branch 
Marcus Hotz 
Director, Program Evaluation 


Treasury Board of Canada, 

Comptroller General 

Evelyn Levine 

Director, Corporate 
Management Services 


Veterans Affairs Canada 
Dick St. John 

Director, Program Evaluation 
Directorate 


Western Economic 
Diversification 

Rod Monette 

Director General, Finance 
and Administration 


990-8090 


996-7245 


992-8565 


990-1825 


998-4625 


954-7881 


954-6041 


993-3633 


994-1224 


992-5148 


990-2670 


951-8291 


956-0890 


1-514-283-6363 


993-4418 


957-2419 


952-3132 


957-2430 


952-2860 


957-7071 


1-902-566-8017 


952-9383 
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This Edition’s Theme: Evaluation of same : 


and Technology Programs 


The theme of this Newsletter is 
science and technology (S&T), a 
continuing priority of program evalu- 
ation within the federal government. 


Alain Barbarie (Energy, Mines and 
Resources - EMR) and William 
Smith (National Research Council - 
NRC) provide an international 
perspective on the subject. They 
discuss a conference held last spring 
on the management of research and 
development, including its evaiuation. 


Next, Benoit Gauthier (formerly at 
Department of Communications - 
COM) provides a departmental view- 
point. He discusses a major evaluation 
of COM’s Research Labs program. This 
study explores a wide range of issues 
and involves an interesting mix of 
evaluation methodology and scientific 
peer review. 


David Moynagh (Office of the 
Comptroller General - OCG) discuss- 
es a recently completed study of S&T 
evaluation, undertaken by the OCG. 
The study reviews S&T evaluation 
work completed since 1984 in 17 
major departments and agencies. 


The Term ‘‘Science and 
Technology’? 


Science and Technology, as defined by 
international convention in the OECD 
Frascati Manual, includes both 
‘‘research and development’ (R&D) 
and ‘‘related science activity’’ (RSA). 


Examples of R&D include basic 
research such as that conducted in 
NRC labs, funding of university 
research by granting councils such as 
the National Sciences and Engineer- 
ing Research Council (NSERC), and 
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research and technology development such as that 
performed in EMR’s Canada Centre for Mineral and 
Energy Technology. 


Examples of RSA include scientific data collection by 
Fisheries and Oceans’ (F&O) Canadian Hydrographic 
Service, testing and standardization by NRC, and other 
forms of data collection and dissemination in support of 
public and private sector activity by Statistics Canada. 


S&T Expenditure Profile 


S&T is performed by over 17 major departments and 
agencies. In all, these organizations will spend approxi- 
mately $5 billion on S&T in 1989-90. The following charts 
provide further detail on the breakdown of expenditures 
by department, i.e. R&D versus RSA, and intramural 
versus extramural performance. Program evaluation is 
operational in each of these organizations. 


| CHART 4: FEDERAL SET | EXPENDITURES 
| 1989- 90 
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Looking at Gross Expenditures on R&D (GERD) in 
Canada as a whole, the federal government is both a 
major performer and a major funder. Estimates for 1989 
indicate it will perform about 17 per cent of GERD and 
will fund 30 per cent. Industry shares are 56 per cent 
and 42 per cent respectively. = 
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W. David Moynagi 


Mr. Moynagh is Chief, Science, Economic and | 
Regional Development with the Program Evaluation | 
Branch, OCG. | 


Economic 
Commission of 
Europe Seminar on age | 
Evaluation in the Management | 
of Research and Development | 


From April 3 to 7, 1989, members of the Economic 
Commission of Europe (ECE) met in Madrid, Spain at a 
seminar on ‘‘Evaluation in the Management of Research 
and Development’’. The seminar was sponsored by the 
Senior Advisers to ECE Governments on Science and 
Technology, a principal subsidiary body of the United 
Nations Economic Commission of Europe. It was attend- 
ed by 26 ECE countries, as well as several international 
organizations. | 


The seminar addressed four themes: 


¢ the organization of evaluation in individual countries 
and in international organizations; 
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the role of evaluation in managing basic research; 


the role of evaluation in managing mission-oriented 
research and national technology programmes; and 


e the evaluation of multi-national research and 
development (R&D) programs. 


Organization of evaluation in individual countries 
and in international organizations 


The papers and discussion of this theme supported the 
concept that the framework for evaluation must reflect 
each country’s political and administrative cultures. This 
| ould vary from a pluralistic, decentralized approach to 
centralized co-ordinated systems, placing legally defined 
rameworks for evaluation across the public sector. 


ton of it was underlined that the approaches to evalu- 


tion of R&D had some points in common: the need to 
sake into account the time and costs involved for 
verforming realistic evaluations; the need for expressing 
dearly the goals and objectives of the research and of 
he evaluations; and the need for interaction between 
on and those being evaluated. 


any countries recognized that their research institu- 
ions were under similar pressures from their govern- 
nents. There is a growing requirement from political 
iuthorities for accountability and relevance of R&D in 
rder to link it more effectively to national economic and 
ocial needs and goals. In a sense, this direction may be 
n conflict with the more traditional evaluation focus on 
juality of R&D. As a result, the growing emphasis on 
iccountability and relevance is having an impact on the 
ountries’ research and development infrastructures as 
vell as on the methodologies for R&D evaluation. 


ncreasingly, R&D evaluation research methods have to 
ncorporate assessments of the relevance of the research 
nd its rationale in light of government policies and goals. 
fast review methods focusing on the quality of the 
cience, and applying traditional peer review methods 
re no longer sufficient. This explains much of the 
juropean interest in Canada’s program evaluation 
Pproaches, and why Canada is seen as an international 
2ader in the practice of the evaluation of R&D programs. 


‘valuation in managing basic research 


he seminar next examined evaluation in the manage- 
lent of basic research programs. The presentations 
xamined a variety of circumstances involving basic 
2search in universities and government institutions. 
valuation was seen to be an important part of basic 
search in many ways: researchers are scrutinized for 
romotion; their submitted manuscripts are screened for 
ublication; and in order to obtain funds for projects they 
1bmit proposals which are reviewed and approved by 
“search funding agencies and foundations. These evalu- 
tion activities rely mostly upon the internal expertise 
f research communities. More recent evaluation efforts 
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differ in that they are commissioned and controlled by 
government agencies and use methods that do not 
necessarily require the expertise of scientists. 


Quantitative methods and, in particular, bibliometric 
analyses are still being used. However, more qualitative 
methods are being used to assess the relevance and 
utility of the research work, as opposed only to its 
quality. These factors imply potential changes in what 
factors are being used in making decisions on R&D work. 


The seminar concluded that evaluation has a variety of 
new and developing applications. Traditional approaches 
and past expectations for the evaluation of R&D need 
to be reassessed in light of these new requirements. The 
conventional meaning of evaluations and peer review 
must not dominate R&D management thinking. Evalua- 
tion data may also draw attention to problems that were 
not seen before, englighten decision makers, or support 
arguments. 


Evaluation in managing of mission-oriented 
research and national technology programmes 


Thirdly, the seminar examined the evaluation of mission- 
oriented research programs and, in particular, national 
programs or national institutions. In the mission-oriented 
R&D, evaluation can fulfill an important feedback role. 
Many participants stressed that this mechanism should 
play an effective role in decision-making and accounta- 
bility for use in long-term, strategic thinking and plan- 
ning. This contrasts with the progress-reporting, reactive, 
information role characteristic of earlier R&D evaluation. 
Close links between R&D evaluation and political 
decision-making were identified in the discussion. The 
assessment covered not only scientific aspects but also 
economic, social, political, regional and ecological issues. 
Notwithstanding the perception that these links exist 
and should be fostered, the difficulty of isolating pre- 
cisely the impacts of evaluations on the decision-making 
process was also recognized. 


During the session, evaluation was seen to have wide 
potential uses, e.g., improving ‘‘fine-tuning’’ of ongoing 
programmes, significant inputs for follow-up of new 
programmes, and increasing the quality of decision- 
making. Nevertheless, it was also stressed that an evalu- 
ation depended fundamentally on the environment in 
which it was performed. Thus, the role of evaluations 
could vary from pure legitimization to a systematic and 


rational basis for decision-making and management 
of R&D. 


Evaluation in the management of international R&D 


The last seminar theme examined the management of 
international R&D programs. This theme is of particular 
importance to European countries, given the increasing 
evaluation studies that are being undertaken on multi- 
national programs. R&D usually has a number of 
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objectives, both tangible and intangible. The intangible 
objectives can be particularly important in international 
collaborative R&D, where, in addition to the achievement 
of specific scientific or technical results, the development 
of transnational relationships can be a prime aim. This 
extra dimension makes evaluation both more challenging, 
given the wider horizons introduced by the intangible 
objectives, coupled with the basic demand for dispas- 
sionate, objective evidence. Papers presented outlined 
several such evaluations. 


Seminar Conclusions 


The seminar reached a series of conclusions on the 
importance and the roles for evaluation in the manage- 
ment of research and development. It recognized that 
the framework for evaluations reflects the various 
political and administrative systems which vary from 
pluralistic decentralized systems to centralized coordi- 
nated systems. Equally, it was felt that evaluation needed 
to reflect the differing existing economic and social struc- 
tures. It was generally agreed that evaluation procedures 
should differentiate between disciplines, and between 
applied and basic research. There seemed to be general 
agreement that both ‘‘top-down’’ (rational or corporate 
perspective on R&D programs) and ‘‘bottom-up’’ 
(scientific, project-based view of R&D programs) 
approaches had their merits. The evaluation should aim 
at increasing the efficiency and effectiveness of research 
and at creating the right conditions for its function. 


Budget constraints, increased R&D costs and growing 
international competition in science and technology are 
all resulting in a growing demand from governments and 
funding organizations for accountability and relevance 
of R&D. As a result, new evaluation mechanisms and 
approaches are emerging which need to undergo a phase 
of institutionalization. At the same time, the greater 
interest of governments and funding organizations in the 
evaluation of basic research is not without possible draw- 
backs. Too narrow and inflexible an approach to evalua- 
tion can be counter-productive to basic research. 


The seminar concluded that the use of the results of 
evaluations as a tool in strategic forecasting, planning 
and management, as well as in assessing effectiveness 
is an important basis for the management of R&D. There 
is a need for research on evaluation and for the exchange 
of experience and cooperation at the international level 
in order to improve the methods and indicators used and 
the institutional arrangements for evaluation, and to 
facilitate the exchange of knowledge on socio-economic 
impacts of national R&D programmes and mission- 
oriented research. @ 


William Smith 

Head 

Program Evaluation Office 
National Research Council 


Alain Barbarie 
Director 

Program Evaluation 
Energy, Mines 

and Resources 


Evaluation 

of Science 

and Technology 
Programs - 

A Departmental 
Perspective 


The following article was submitted by Benoit Gauthi i: 
Vice-President, Ekos Research Associates Inc., formerl 


1 


with the Department of Communications of Canada 


The Department of Communications pursues man 
varied objectives, but its activities all serve its centre 
mission, which is: ‘‘Nation building - by helpin. 
Canadians to share their ideas, their information ant 
their dreams’’. The Department is committed to ensurin!. 
that Canadians enjoy the best communications service 
in the world and the largest possible choice in the area 
of cultural expression and information technologies. M 
achieve its goals, the Department employs a variety 0. 
means, including scientific research in the fields 0! 
telecommunications technology, specifically, radio ant 
satellite communications; communications equipment 
and instruments, i.e. microelectronics and fibre optics| 
workplace automation; broadcast and video technolo: 
gies; and, until recently, aerospace technologies. 


In 1988, the Assistant Deputy Minister responsible for 
the Research Sector asked the Director of Program 
Evaluation to move up the evaluation work in this sec- 
tor in relation to the five-year planning. Having only 
recently assumed the position, he felt that he had no 
solid factual analysis to enable him to make an enlight- 
ened judgement on the performance of this sector of the 
Department. He hoped that the evaluation of its 
programs would provide him with such a PERD 


| 
The important issues for the Assistant Deputy Ministae 
were clear: Is the work done by the laboratories rele- 
vant? Is it of high quality? How does it compare with the 
work done by other laboratories in Canada and around 
the world? In light of the government’s major priorities, 
the literature on the subject and other S&T evaluations, 
the evaluator added issues that are usually associated 
with evaluations performed in accordance with the 
standards of the Office of the Comptroller General: Are 
the laboratories aware of the needs of their clients? Are 
their findings disseminated widely enough? Does the 
government have a rational justification for maintaining 
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public laboratories? In the end, the Deputy Minister 
approved the following list of evaluation issues: 


L Is there a plausible connection between the labora- 
tories’ objectives and activities? 


° To what extent are the research laboratories geared 
| to their clients’ needs? 


¢ How widely are the laboratories’ findings 
disseminated? 


e What is the level of quality of the laboratories’ output? 


e Is it essential that the Department of Communications 
own and operate communications laboratories in 
order to achieve its mission? Are laboratories a good 
investment from an economic standpoint? 


e Could some activities be carried out more effectively 
by the private sector? 


All these questions had to combine both an internal 
approach and a comparative perspective, in relation both 
to other Canadian laboratories and to foreign laborato- 
ries. The issues seemed to suit all the players concerned, 
including the Assistant Deputy Minister, the Deputy 
Minister, Treasury Board and the Privy Council Office. 


Obtaining objective, factual, systematic information on 
all these issues was to prove a long and difficult task, 
full of obstacles. First it was necessary to break the job 
down into studies of reasonable size that could be 
contracted out to consultants - the modus operandi of 
the Department’s Evaluation Division. The Division 
maintains complete control over initial planning of the 
evaluation, doing the work related to the evaluation 
assessment report. Then it contracts out the work 
equiring a major investment of manpower, while 
etaining responsibility for the final strategic integration 
of the findings and drafting the recommendations. The 
‘ollowing studies were carried out: 


» opinion survey of researchers from other countries 
who have worked on assignment in departmental 
laboratories for several months; 


survey of the laboratories’ clients; 
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study of the availability of research reports within the 
Department; 


ns 


analysis of the quality of the laboratories’ output by 
external peers; 


analysis of client-orientation of the laboratories over 
the years; and 
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critical analysis of the costs and benefits of selected 
laboratory activities. 
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We would point out that bibliometric techniques were 
not among the research methods used in this evaluation. 
It has been established that, while such methods are use- 
ful in some settings (especially universities), they are not 
appropriate in the context of public laboratories 
managed in the manner of those of the Department of 
Communications. In fact, there is little incentive for 
public-laboratory researchers to publish. The valuable 
activities would not be ascertained by methods based on 
bibliometrics. 


Intuitively, using cost-benefit analysis seemed rather 
common place. Wouldn’t it be useful to show how much 
economic activity the laboratories’ research generates? 
What amount of economic benefit can be attributed sole- 
ly to the activity of the laboratories? If such benefits 
equal the public investment, isn’t this a useful justifica- 
tion for the laboratories’ existence? However, it was 
found that demonstrating this is much more difficult 
than expected. The data are vague, it is almost impossi- 
ble to determine the contribution made solely by the 
laboratories, the amount of time needed to observe an 
effect makes the imputation of causes improbable, the 
benefits are difficult to account for, and so on. Neverthe- 
less, our work has made it possible to determine a field 
of values within which the actual economic effect of the 
laboratories is probably situated. Recognized internation- 
ally as difficult to apply in the case of S&T programs, this 
focus nevertheless has its place in the range of pertinent 
evaluation questions for such activities. 


Special emphasis has been placed on two of these 
studies: the client survey and the quality analysis. In the 
past few years, the laboratories have sought to improve 
their relationship with Canadian industry. It was there- 
fore particularly important to judge the results of these 
efforts and to provide information useful for making stra- 
tegic decisions in this respect. Moreover, it was 
acknowledged that the laboratories would be able to 
prove their relevance only to the extent that they rated 
very favourably as regards to the quality of their work. 
Both for the client perspective and the question of qual- 
ity, an international perspective was necessary, since in 
many respects the industry is trans-national in nature. 


It should be noted that this particular emphasis on the 
quality of the scientific work does not correspond with 
the operational trends in evaluation observed elsewhere 
in Canada and in other OECD countries. In fact, in most 
cases, the review of quality by peers is divorced from 
evaluation of a more sociological and economic type. We 
firmly believed, in planning the evaluation research, that 
the examination of quality had to be conducted in 
tandem with the gathering of clients’ opinions and the 
analysis of the laboratories’ economic efficiency. It is 
unlikely that a laboratory can be called a success if it 
does not have, at least partially, these three 
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characteristics. For example, what would a laboratory 
that did not produce quality outputs be worth, even if 
its products were considered useful by its client compa- 
nies? The integration of peer assessment of product qual- 
ity with the evaluative process was an important factor 
in enhancing the credibility of the results of this 
evaluation. 


The challenge of international comparison was handled 
differently in these two studies. In the client survey, the 
Canadian respondents were asked to compare the 
Department’s laboratories with other Canadian and 
foreign laboratories that they knew. This comparison was 
performed without difficulty. Moreover, the research 
design required that the firm interview a sampling of 
clients of two foreign laboratories in order to provide a 
basis for comparison. This operation was not as success- 
ful. One of the laboratories simply refused to participate, 
while it took a great deal of negotiating skill to receive 
less than complete cooperation from the other. The 
product-quality study was limited to asking the peers to 
compare the product with products of the same type 
from other laboratories. The peers also had to judge the 
quality of the overall output of the laboratory from 
which the product in question came. Lastly, they were 
asked to indicate what laboratories they were thinking 
of when they made their comparisons. This approach was 
more successful than that of the control group used in 
the client survey. 


In conclusion, it is worth recalling that several sources 
of information are better than one. It seems that the 
evidence provided by the above-mentioned studies con- 
verge to a large extent, thus confirming each other. Since 
the weaknesses of one study are often offset by another, 
parallel study, we can be confident that the final results 
probably constitute a faithful representation of 
reality. # 


Benoit Gauthier 


Evaluation of 

science and Technology 
Programs in the 

Federal Government 


This article provides an overview of the context for t] 
OCG’s S&T evaluation study. It also describes th 
approach to and results of this project. Some initi 
thoughts on the implications of the study results are al) 
discussed. | 


Context 


The study examined the nature of S&T evaluation wok 
produced since 1984 by major departments and agencie 
It also analysed the uses of this work in relation to tl 
accountability and decision-making processes 
departments and the government. The data and info 
mation gained from this project provide a good basis fro 
which to document and assess what has actual) 
happened, and to improve the relevance and usefulne! 
of S&T evaluation in the future. | 


The research project was inspired by two main factor 
First, it was timely to review the appropriateness of th 
1986 OCG guidelines, Evaluation of Research an 
Development Programs, based on experience gained 
Second, there had been several recent developments i 
federal S&T policy and management that needed to b 
taken into account in planning and conducting S& 
evaluations. These developments included: the federe 
government’s S&T strategy (InnovAction); its manage 
ment structure and process (S&T Decision Framework) 
and, related policies and decisions on Major Surveys 
Technology Centres and S&T Extramural Performance 


Approach 


To assure reasonable reliability of data and depth 0 
analysis, the design of the S&T research study consistec 
of three intersecting lines of enquiry. 
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¢ First, 61 S&T evaluation studies completed since Aga 
1984 were summarized. Actions taken as a result 0! 
these evaluations were also documented. This line ot 
enquiry provides the major source of empirical date 
on the nature and use of S&T evaluation work. 


e Second, surveys were conducted of S&T evaluation 
practitioners, S&T managers and central agency 
analysts concerned with science and technology. In 
all, 33 people in 15 organizations were interviewed. 
The survey findings help to identify the management 
context for S&T and to examine the nature and use 


of evaluation in this regard. 


e Third, a review of the relevant literature was 
completed. This helps to situate the scope, method- 
ology, findings and results of the overall study. 
Canadian and international literature on S&T, 
S&T evaluation and the utilization of evaluation were 
examined. 


Findings 
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S&T Evaluation Coverage 


Inthe major S&T departments, there are approximately 
160 separate program evaluation components either fully 
dedicated to S&T or containing significant S&T elements. 
About two-thirds of these components have been wholly 
or partially evaluated, mainly in the last five to six years. 
In all, 123 individual S&T evaluation studies have been 
‘completed. 


Chart 3 shows the distribution of completed studies by 
fiscal year with projections to 1991-92. 
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Source: OCG/PEB Database & Departmental 
Evaluation Plans, September 1989. 


S&T Evaluation Issues 


The survey findings indicate a growing interest in having 
S&T evaluations address program rationale and cost- 
affectiveness issues. This trend is also becoming more 
2vident in recent international literature on S&T evalu- 
ition. The emphasis on these issues reflects optimism 
that evaluation has matured to a point where it can take 
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on more complex tasks. However, this optimism needs 
to be tempered. Our review of S&T evaluations indicates 
that, while rationale and cost-effectiveness issues are 
prominent, they are not often examined in the kind of 
depth that current and projected demands would appear 
to require. 


Our surveys and the literature indicate that interest in 
objectives achievement and impacts/effects issues is more 
common for mission-oriented S&T than for basic research 
activities. In one sense, this finding is not surprising, 
given that basic research is often described as goal-free 
and that there is a long-range time horizon before speci- 
fic impacts/effects might be expected to appear. From 
another perspective, however, the fundamental objec- 
tive of basic research is scientific excellence, and 
certainly this is an aspect of program performance worth 
examining. This apparent contradiction probably is 
explained by the fact that the most common means of 
assessing quality of scientific research is peer review, a 
technique which has been applied by scientists them- 
selves, generally with a focus on individual R&D projects 
and done outside of the program evaluation process. 


This may be changing. For example, peer review was a 
key aspect of NSERC’s Strategic Grants program evalu- 
ation. In this issue, Mr. Gauthier describes a major evalu- 
ation involving a significant peer review element. He 
argues (correctly I believe) that the quality of scientific 
research is an almost inseparable aspect of R&D perfor- 
mance. In this context, he also argues that peer review, 
although crucial, is not in itself a sufficient means of 
getting at the issue of quality. Recent S&T evaluation 
literature and guidelines coming from the OECD also 
encourage a closer integration of peer review of science 
within the scope of evaluation. 


Other issues, particularly of interest to federal S&T 
conducted ‘‘in-house’’, include the working environment 
and extramural performance. In the first case, the 
emphasis is on the extent to which the working environ- 
ment of the bench scientist supports professional 
development, stimulates creative thinking, fosters scien- 
tific excellence and provides the necessary equipment 
and facilities to support R&D work. In the second case, 
extramural performance of S&T (e.g., contracting-out, 
joint ventures, etc.) is seen as a key mechanism for tech- 
nology transfer, for encouraging the investment of out- 
side resources in federal S&T and for promoting an 
increased level of R&D activity within Canada. 


Finally, interest in focusing S&T evaluation on improve- 
ment in program operations and delivery was very 
pronounced in all three lines of enquiry. This follows the 
trend for program evaluation in general. 
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Chart 4 shows the percentage of evaluation studies 
which sought to address each of the four basic evalua- 
tion issues: program rationale, objectives achievement, 
impacts/effects and alternatives. 


CHART 4: ISSUE FOCUS OF S&T EVALUATION ~ 
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Source: OCG Study on S&T Evaluation, 
September 1989 (n-61). 


S&T Evaluation Methods 


Commonly-used evaluation methods have been surveys 
of client and expert opinion, which is understandable, 
given the preeminence given to client relevance and use 
in most federal S&T. However, our analysis of completed 
S&T evaluations reveals the need for a stronger ‘‘multiple 
of evidence’ approach that goes beyond identifying 
levels of client satisfaction and gets to the heart of the 
more gritty issues of program need and effectiveness. 


As noted earlier, peer review is an available, though 
seldom used, technique in federal R&D program evalu- 
ation work, although the nature of some programs is 
such that the clients and study team experts can some- 
times serve as a sort of peer group. Recent evaluation 
initiatives and literature suggest that more could be done 
to integrate peer review techniques within program 
evaluation methodology as it applies to R&D activities. 


The application of socio-economic impact analysis, cost- 
effectiveness analysis and benefit-cost analysis in S&T 
evaluation encounters the same general difficulties as 
it does in other evaluation, e.g., isolating the chain of 
effects, measuring incrementality, etc. A particular 
problem cited for S&T is that the time horizon for 
observable impacts can be very long. Despite these 
difficulties, the recent literature on S&T evaluation 
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supports the use and continued refinement of these tecl 
niques. A few of the more recent federal evaluatior 

have experimented with ways to apply these methoo ‘7 
more successfully. 

Internationally, bibliometric and citation analysis ar 

commonly practiced methods in assessing the quality ¢ 

R&D. However, the analysis of federal evaluations show 
virtually no use of these techniques. 


Chart 5 shows the various data collection/analysis 
methods used in the S&T evaluations reviewed. 


CHART 5: S&T EVALUATION DATA : 
COLLECTION MÉTHODES 
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Source! OCG Study on S&T Evaluation 
September 1989 (n= *61). 


Uses of S&T Evaluations 


The literature review uncovered very little empirical 
analysis of the use of evaluation and, in particular, S&T 
evaluation. There is reasonably extensive literature on 
general evaluation utilization. 
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In our survey, a number of S&T managers cited the 
involvement of evaluators, in itself, as an important 
element in the utilization of S&T evaluation. Managers 
acknowledge that in some cases evaluation studies have 
been overly positive. However, their general view is that | 
the work of evaluators is neutral and objective. This 
increases managers’ level of comfort in using the findings 
and results of the evaluation work. | 

| 


The analysis of completed S&T evaluations and the 
survey findings indicate that modification to program 
design and operations has been a main use of S&T 
evaluation. 
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| There are several notable examples where S&T evalua- 
tion has had a significant impact on policy. For example, 
_NSERC’s evaluation of its Strategic Grants Program was 
a major factor in defusing concerns regarding the 
relevance and worth of this program. Also, the combi- 
nation of several of Agriculture Canada’s evaluations 
helped S&T management to define a new policy focus 
for the department’s Research Program. In many other 
cases, S&T managers report that this type of impact is 
indirect in the sense that it has not necessarily come 
from specific recommendations. Rather, it comes from 
an overall impression built up by the evaluation report. 
And frequently, it becomes evident only well after the 
completion of the evaluation itself. 


Many S&T managers see evaluation as an important 
‘means of obtaining information on client characteristics 
‘and perceptions. Action to improve consultations and 
communications with clients is a frequently cited impact. 
‘Conversely, some managers commented that the results 
of client surveys tended to be too congratulatory (what 
some observers have referred to as ‘‘grateful 
testimony’’). 


S&T managers appeared less inclined to find alternatives 
or cost-effectiveness analysis useful because of the either 
_ or, conversely, summative treatment these 
issues often received in program evaluation work. 


Chart 6 shows the different uses to which the findings 
of completed evaluations have been put. 


‘CHART 6: USES OF S&T EVALUATIONS 
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Source: OCG Study on S&T Evaluation, 
September 1989 (n-50). 
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Further analysis of the nature and use of these 
61 completed S&T evaluations is being done and this will 
be presented in the study's final report. 


Implications of the Study 


The OCG will be consulting with the S&T evaluation 
community, and others, concerning the findings and 
consequences of this study. Minimally, the following 
actions seem called for. 


e The guidelines for S&T evaluation should be revised 
to reflect the current S&T management process and 
demands for information, to take better account of 
the heterogenous nature of S&T programming and to 
take advantage of evaluation experience gained. They 
should be formatted as work instruments for evalua- 
tors and not presented as general discussion papers. 
The OCG needs to do a much better job of promoting 
awareness and use of these guidelines in departments 
and at the centre. 


¢ This effort should be supported by periodic seminars 
and workshops to allow for sharing of perspectives 
and experience. These sessions would be principally 
aimed at S&T evaluators, but could, from time to time, 
include S&T managers and central agency analysts. 


+ A small standing committee of senior S&T evaluators, 
S&T managers and central agency analysts should be 
formed to provide ongoing advice on the development 
of evaluation in this area. @ 


W. David Moynagh 


Mr. Moynagh is Chief, Science, Economic and Regional 
Development with the Program Evaluation Branch, 
OCG. 


Improving or Verifying 
Cost-Effectiveness 


In addition to the measurement of the achievement of 
objectives and other effects, the scope of program evalu- 
ation in the federal government includes the examina- 
tion of rationale and alternatives. Several recent 
developments mean that the latter issues are being given 
more recognition. 


Spending restraints are continuing and even intensifying. 
Therefore departments must be sure they are providing 
only sensible programs and that their delivery is effi- 
cient. In fact, reallocating from program terminations or 
efficiency savings is practically the only method 
remaining to create funding flexibility. Under IMAA 
agreements, departments will need to prove the worth 
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and efficiency of their programs. With the encourage- 
ment of the Public Accounts Committee of Parliament, 
the Office of the Auditor General is taking a special look 
at the management of efficiency over the next few years. 
The new Bureau for the Delivery of Government Services 
in Treasury Board Secretariat is encouraging depart- 
ments to identify areas for restructuring, i.e. non- 
traditional alternatives to program delivery. 


With all this interest, internal and external, it seems that 
most, if not all, evaluations should be incorporating 
serious work on rationale and alternatives. To what 
extent is this happening? What is the quality of these 
investigations? The OCG has been examining these ques- 
tions and intends to provide policy and methodological 
guidance. Our first research indicates significant activity 
along these lines is occurring, but that there is probably 
substantial scope for more. 


Some of the tasks we have set for ourselves this year 
include: 


+ further research on the occurrence and quality of 
rationale and alternatives investigation in recent and 
ongoing evaluations; 


¢ a paper on how to verify program rationale and 
develop and test alternatives; and 


e¢ a workshop for evaluators on this topic (scheduled for 
March of 1990). = 


John Batchelor 


Mr. Batchelor is Senior Policy Analyst with the Program 
Evaluation Branch, OCG. 


International Working Group on 
Program Evaluation 


The International Institute for Administrative Sciences 
(AS) is an international non-governmental organization 
which serves as a clearing-house for international infor- 
mation on administration. Currently, it is the only world- 
wide organization which aims at providing 
interdisciplinary and comparative studies of inter- 
national, national, regional and local public adminis- 
tration. This is done through its newsletter, Interadmin, 
its journal, the International Review of Administrative 
Sciences, published in both English and French, and 
through the publication of reports prepared by various 
research committees and working groups. 


Founded in 1930, the ILAS has approximately 100 current 
members from Europe, Asia, North America, the Arab 
World, Africa and Latin America. These members may 
be states (such as Canada), national sections (such as the 


Institute of Public Administration of Canada) 
international governmental organizations (such as the 
International American Development Bank), corporate 
members, or specialized organizations (such as the 
European Group of Public Administration). . 


In February 1986, the Research Committee on Adminis: 
trative Structures and Management of the IIAS 
announced the creation of a Working Group on Policy 
Evaluation. The Research Committee established rather 
extensive terms of reference which ranged from a 
description of current practice among member states to 
issues of utilization, methodology, the relationship of 
evaluation research to budgeting, auditing and other! 
disciplines and a concern for meta-evaluation. 
Dr. Ray Rist of the U.S. General Accounting Office) 
assumed the responsibility of chairing the Working Group 


and calling its first meeting. | 


The first meeting took place at the offices of the IAS: 
in Brussels in November 1986. Canada, the U.S.A., five! 
Western European and three Scandinavian countries 
were represented. Our major task was to develop an 
agenda which would guide our work for the next several 
years. 
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The Working Group presented its first substantive’ 
product, National Reports, at the 1987 Paris meeting. | 
These documents described current evaluation policy 
and practice found in the various states represented at 
the meeting. Following review it was agreed the National 
Reports should be revised for future publication. 
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The third meeting of the Working Group began with a | 
series of studies on utilization of evaluation research. A | 
wide variety of European and North American perspec- | 
tives were presented, reviewed and subsequently | 
revised. Two other important developments occurred at | 
the Leiden meeting. The first was to encourage the” 
participation of more public servants in our work. The. 
second was to form three distinct sub-groups which 
would prepare a more elaborate agenda for the 1989. 
meeting in Denmark. | 


At Aarhus we were delighted to find that public servants | 


from Canada, Sweden, the Netherlands, Finland and the 


U.S. had accepted invitations to participate. Canada’s 


representation was enhanced by the addition of 


John Mayne from the Office of the Comptroller 


General. The sub-group meetings produced some rather. 


intense discussion before agreement on the Working 
Group’s research agenda for the next few years took" 
place. We expect rather substantial reports on evalua- — 


tion, budgeting and auditing, personnel policy, and utili- 
zation from these subcommittee members at the 1990 
meetings in Stockholm. 


The Working Group on Policy Evaluation has provided 


a forum in which academics and public servants 
representing approximately a dozen different states can 
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meet and discuss their common interest in evaluation 
research. In addition, some of the work prepared by 
members of the Working Group will be published shortly. 
Dr Rist has edited Program Evaluation and the 
Management of Government which will be published this 
fall by Transaction Press. It includes the National Reports 
of the various states represented on the Working Group 
as well as a comparative analysis prepared by 
Hans-Ulrich Derlien of the Federal Republic of 
Germany. The utilization papers presented at the Leiden 
meeting will be published in late 1989 or early 1990 in 
Knowledge in Society. 


ithe Working Group also sponsored a panel at the annual 
merican Evaluation Association Conference in San 
Francisco this October. 


is interaction between academics and public servants 
is useful. On the one hand, practitioners within the 
public service criticize the naivety and occasional ‘‘ivory 
tower”’ arguments of the academics. On the other hand, 
academics decry the lack of methodological vigour, the 
failure to use evaluation findings and the emphasis on 
process evaluation which are, perhaps, too-common 
characteristics of evaluation research within govern- 
ment. Maybe the tension and communication between 
the two groups will result in reports and publications 
which reflect the need to strike a healthy balance 
between the two points of view. m 
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R.V. Segsworth PhD 


Dr. Segsworth is Associate Professor, Department of 
Political Science at Laurentian University, Sudbury, 
Ontario. 
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Canadian Evaluation Society 
Annual Conference 1989 


The Canadian Evaluation Society held its 10th annual 
conference from June 29 to 31, 1989, in Fredericton, on 
the theme ‘‘Evaluation: Towards Accountability’. More 
than 280 persons, from all regions of the country, 
attended the conference. This success is considered a 
concrete sign of the Society’s maturity. 


The Conference Committee was chaired by 
Harriet Lenard, senior policy analyst with the Policy 
Secretariat, Government of New Brunswick. She was 
assisted by a local organizing committee and a program 
and workshop committee. The following federal evalua- 
ors were on the latter committee: William Baker 
RC-T), Daniel Caron (INAC) and Sheila Faure (NPB). 


After the first day, which was devoted to workshops, the 
Premier of New Brunswick, the Honourable Frank 
McKenna, officially welcomed the participants. The 
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Premier noted the role of evaluation in resource alloca- 
tion and the fact that politicians must undergo evalua- 
tion every time they face the electorate. 


After the customary thanks, the President of the CES, 
Zena Vigod, invited Ross Connor, President of the 
American Evaluation Society, to say a few words. 
Mr. Connor praised the Canadian contribution to the 
field of evaluation. 


John Sloan, Secretary of the Management Board of the 
Ontario Cabinet, was the guest speaker at this plenary 
session. After pointing out the importance of evaluation 
to effective management, along with the dynamism of 
the CES’s Ontario chapter, he expressed his views on the 
relationship between accountability, program evaluation 
and human beings. He presented the four principles that 
have guided his organization since a more formal evalu- 
ation approach was introduced: (1) co-ordination of 
studies; (2) selection of the programs to be evaluated; 
(3) planning of studies; and (4) the obligation to develop 
an evaluation policy for departments. 


The following federal evaluators and officials contributed 
to the various workshops and sessions: (Agr) Ian Kelley, 
Carol Motuz and Bernice Vincent; (CCAC) Kenneth H. 
Tiedemann; (CMHC) Philip Brown; (CSC) Gordon 
Farrell; (Com) David Black and Benoit Gauthier; 
(EMR) Alain Barbarie and Charles W. Moore; (HWC) 
Bob S. Allen, Judy Berger, Michel Laurendeau, Bud 
Long, Ed Overstreet, Sayed Shelbaya and Carol 
Turriff; (INAC) Daniel J. Caron, Bev Clarkson, Allan 
Gordon and Alan Winberg; (Lab) Bonaventure Bowa 
and Iris Bradley; (NPB) Sheila Faure; (NRC) William 
Smith and George Teather; (OAG) Maria Barrados, 
Tom Beaver, Neil Maxwell, Hugh McRoberts, Nancy 
E. Stoneman and Paul Ward; (OCG) Ronald C. Corbeil, 
Guy Leclerc, Cameron McQueen, David Moynagh and 
Toru Uno; (PSC) Avrim D. Lazar; (RC-T) William 
Baker; (SGC) P.J. Carr and Stan Divorski; (SSC) Celine 
Cheah, Adéle Copti-Fahmy, Louise Durocher and 
Jennifer Eunice Stewart; (TC) Norm Steinberg. 


Topics were varied, ranging from evaluation in Crown 
corporations and various discussions on accountability, 
to the contribution of consultants and future consider- 
ations. Some case studies added extra flavour to the 
proceedings. 


At the CES’s annual general meeting, which took place 
as part of the conference, all the items on the agenda 
were reviewed. It was learned that the number of 
members has passed the 1,000 mark and that the 
Canadian Journal of Program Evaluation and our 
newsletter are doing quite well. The regional reports 
were very positive, and the Society is actively working 
to establish provincial chapters where none yet exist. 


Lastly, Zena Vigod presented a positive assessment of 
her term as president of the CES, noting that progress 
had been made on all priorities (clarifying the roles and 
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responsibilities of the regional and national entities, 
improving services to members, increasing the level of 
professional development, instilling dynamism in the less 
active regions, developing and initiating a strategy to 
increase the CES’s visibility, increasing sources of 
revenue, and keeping the individual membership fee at 
$55). Our, congratulations to Ronald C. Corbeil (OCG) 
on his appointment as CES president! 


The day after the lobster feast, the participants were 
invited to attend a conference luncheon at which 
Arthur Doyle of the University of New Brunswick spoke 
on the theme of changes in the political world. He noted 
the increasingly important role of the media, lobby 
groups and opinion surveys. At that event, John Burns, 
Chairman of the Awards Committee of the National 
Capital Chapter (NCC) of the CES, presented the NCC’s 
Student Evaluation Essay Contest award to Pamela 
Brett of the University of Guelph. The award included 
a $500 cash prize and Ms. Brett’s expenses at the Annual 
Conference in Fredericton. Her essay was entitled 
“Evaluation of an Academic Support Program for High 
Risk University Students’’. Ms. Brett is the first recipient 
of this award. Guy Leclerc, Deputy Comptroller General 
for Canada, presented the CES Public Sector Award to 
Mark Cox, Director, Government Activity Review, 
Ontario Management Board of Cabinet for his work in 
designing, developing and establishing the evaluation 
function in the Government of Ontario. In making the 
presentation, Mr. Leclerc noted the high quality of 
Mr. Cox’s work. Mr. Cox is the first recipient of this 
award. Mr. Corbeil, President of the CES, presented the 
CES Award for Contribution to Evaluation in Canada to 
Ron Robinson, President of Abt Associates and to 
John Mayne, Director, Evaluation Policy, OCG. 
Mr. Corbeil noted the exemplary contributions that each 
has made to the field of evaluation. This award has been 
presented only three times in the history of the CES. 


From left to right: CES president, Ronald C. Corbeil with recipients 
of CES Award for Contributions to Evaluation in Canada, 
John Mayne (OCG) and Ron Robinson (Abt Associates) 


Mark Cox, Director, Government Activity Review, Ontaraal 
Management Branch of Cabinet, recipient of the CES Public Sector 
Award 


| 
| 


A plenary panel discussion had been organized to close 
the conference. New Brunswick’s Minister of Commerce | 
and Technology, the Honourable A.W. (Al) Lacey, | 
emphasized the importance of the relationship of trust, 
that must exist where authority is delegated, and the 
need to create an environment in which there are no 
surprises. In his opinion, a minister is accountable to the. 
premier, the caucus and, ultimately, the electorate. 


Claire Morris, Deputy Minister, Policy Secretariat, 
Government of New Brunswick, approached accounta- | 
bility from the standpoint of deputy ministers. She saw ! 
evaluation as the government’s main tool for ensuring | 
that its objectives are achieved effectively and effi. 
ciently. She considered evaluation to be especially | 
important in a context of budget restraint and when a | 
new government takes office. 1 


Ed Graham, Vice-Chairman of Brunco Inc., explained — 
the accountability and evaluation process used by his — 
organization. The process involves five phases: (1) setting 
objectives; (2) gathering information; (3) calculating the 
gap between performance and expectations; 
(4) analysing variances; and (5) reconsidering the organi- 
zation in the light of the results obtained in the previou 

phases. 1 
À 


To conclude, Ken Ross, Executive Director of the Ne 
Brunswick Division of the Canadian Mental Healt 
Association, saw evaluation as an important manage 
ment tool whose key word is trust. 

The 11th conference of the Canadian Evaluation Society 
will be held May 13-16, 1990, in Toronto. You are 
invited and we hope to see you there! = 


oo 


Hervé Déry 


Mr. Déry is Evaluation Analyst with the Program 
Evaluation Branch, OCG. 


bo. you a an interesting hicies on program 
i evaluation that we can publish? The Newsletter 
- welcomes | all contributions . in either official 


anguage | from the evaluation community. For 
further information, please call the Editor, 
Paulette Somerville at (613) 957-7167. à 


Classifying Evaluation Use 


Elsewhere in this issue is an article on a study of evalu- 
ation of Science and Technology Programs. Utilization 
was one of the principal topics and several examples are 
discussed. The results of most recent S&T evaluations 
have been categorized and included, along with others, 
in the Scoreboard for this issue. Once again, the pattern 
of use seems to be fairly stable. Just about all evaluations 
get used in some way and a solid majority play some part 
in concrete improvements or changes to the evaluated 
programs. 


Program Evaluation 


There are many ways to classify evaluation use. For the 
S&T study, the classification of evaluation consequences 
used was more detailed than the one shown in the Score- 
board. A reconciliation was necessary before those 
studies could be incorporated. Similarly, data on utiliza- 
tion discussed in the 1989 Government Program 
Evaluation Plan (GPEP) reflect a slightly different 
classification scheme. There, the categories were not 
exclusive and some of the same terms were defined 
differently. 


This variation seems frustrating at first glance but is 
really inevitable. It is related to the questions being asked 
in the research. In our Newsletter Scoreboard, the most 
important points are whether evaluations are being used 
at all and whether programs change. It would take a 
different categorization to illustrate the degree of causal- 
ity or attribution, for example. The GPEP data were 
gathered to show, among other things, whether evalua- 
tions were changing the external aspects of programs - 
how the clients see them - rather than merely provok- 
ing internal economies and adjustments. 


One aspect of utilization not discussed so far is account- 
ability. The OCG hopes to take a look at this during the 
fall. Utilization of S&T evaluation, utilization classifica- 
tion and accountability are three of the topics that likely 
will be on the agenda of a workshop that PEB and the 
National Capital Chapter of the CES are organizing, 
probably for December or January. Details will be 
announced as arrangements become firmer. Please call 
John Batchelor (957-7179) if you would be interested 
in making a presentation on a particular case or on the 
general theory of utilization. 


Another interesting classification of evaluation use, and 
incidentally an external comparison, surfaced recently. 
Murray Elston, Chairman of the Management Board in 
the Government of Ontario, told an Ontario CES confer- 
ence that 50% of evaluations in the provincial govern- 
ment were confirming the status quo while the rest led 
to terminations or substantial program changes. These 
results appear a little less active (in the sense of effec- 
tive changes) than the federal results but there are 
several possible explanations for the difference. First, 
Ontario programs, or at least the ones they chose to 
evaluate, might be less in need of improvement than 
federal ones. Second, the Ontario definition of ‘‘substan- 
tial changes’’ could be more restrictive than the ‘‘modifi- 
cation’ category used in the federal scoreboard. They 
might count some of our more modest changes as 
confirmations. Third, differences in scope, experience, 
methods and managerial environment could affect the 
use of evaluations in the two governments. Nevertheless, 
it is encouraging to see evaluation playing a substantial- 
ly positive role in another jurisdiction as well. # 


À" ——" —"——————…————————————— 
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NS 
UTILIZATION SCOREBOARD 


Evaluations documented 84.04 to date 


Instrumental Use 
Termination 
Reform 
Modification 
Confirmation 


Conceptual Use 
Increased Understanding _ 


No Results 


The above program consequences were associated 
with, but not necessarily solely attributable to, 
completed program evaluations done i in 30 major 
departments and agencies. 


Termination is used where evaluation played a 
part in the decision to close down a program. _ 


Reform describes a study’s assocation with major 
changes in the design or scope of a program. 


Modification indicates that more modest changes — 
were implemented to improve efficiency and/or 
effectiveness. 


Confirmation captures those situations where the 
evaluation figured in an active decision not to 
change the program. This includes the extension 
of a sunsetted or pilot program. 


Increased understanding indicates that some 
new information about the program was discovered 
and communicated which may eventually influence 
policy or program decisions. 


No results - this category is reserved for studies 
which produced no valuable information or which 
were never reported, perhaps because a program 
was cancelled, transferred or reorganized before 
the study was wrapped up. 


John Batchelor 


Mr. Batchelor is Senior Policy Analyst, Program 
Evaluation Branch, OCG. 


PED’s Appearance Before House 
of Commons Standing Committee 
on Agriculture 


In May 1988, representatives of the Program Evaluatio 
Division (PED) of Agriculture Canada appeared a! 
witnesses before the House of Commons Standinj 
Committee on Agriculture. As far as we know, this wa: 
the first time that a government evaluation unit hac 
appeared before any Parliamentary Committee. At the 
time, the committee was studying options for financia 
restructuring of farm debt, and had invited the PED tc 
discuss the results of its evaluation of the Farm Debi 
Review Boards, one of several initiatives Agricultur 
Canada had taken in the face of the rising crisis ol 
Canadian farm debt. | 


Those appearing before the Committee were. 
George Paterson, Director General of the Audit and 
Evaluation Branch; Carol Motuz, Acting Director of 
Program Evaluation, and Bernice Vincent, Program 
Evaluation Officer. 


Instructions from the standing committee were clear and 
simple - we should open with a statement concerning) 
the evaluation function and the evaluation findings, to! 
be followed by an open question period by the members! 
of the committee. As a first step in our preparation, we 
contacted the Minister’s Office to discuss with his staff 
the parameters of our presentation. This step was both. 
necessary and useful, since the program environment, | 
and thus evaluation, included various high visibility 
issues, such as the ‘‘farm debt crisis’ and the future of 
the family farm. Our evaluation study had just completed’ 
the data collection stage, and conclusions and recom-| 
mendations, which might later lead to policy, had not! 
yet been formulated or approved, so it was important! 
to concentrate on the findings of the evaluation. 


Our opening statement focused on the evaluation func- | 
tion in general; the approach to evaluating the Farm Debt 
Review Boards (which included in-depth interviews, | 
primarily with public and private lending institutions, 
focus groups in farming communities, and a client survey 
of 4,200 clients of the Review Boards); and the results | 
of the evaluation. | 


Questions by Committee members reflected their 
concerns over the powers of the Boards (Were these 
sufficient for dealing with cases of insolvency?); the 
possible need for further measures in support of the 
farmers; the views of the banks; the state of farm. 
consultancy and the quality of advice available to 
farmers; the need for better bookkeeping by the 
Canadian farm community; and the cause and the extent 
of the farm debt crisis. 
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Some direct quotes follow which illustrate Members’ 
acute understanding of the situation, as well as their 
sensitivity to farmers’ needs: 


Mr. Althouse: ‘‘Did you get any sense that the 
creditors’ concern about falling land values is also 
the concern of the panelists and the board members 
who may not be terribly interested in seeing land 
values drop since most of them have a lot of their 
assets tied up in land as well?” 


Mr. Scowen: ‘‘A complaint I got was the farmer felt 
deflated. First he went to the debt review board 
and felt he was getting a deal he could handle. He 
was prepared to carry on with his plans for seeding, 
or whatever he was doing, only to be shot down 
a month or two later by Farm Credit Corporation. 
I wonder if your survey showed that.” 


Mr. Brightwell: ‘‘Does the report you are going to 
circulate indicate those areas where it (the state of 
agricultural consultancy) is deficient in 
Canada? . . .If it does not, I would like to know. Is 
cost a major part of every farmer not having an 
advocate on his side?’’ 


PED was very pleased to have such direct input into an 
integral part of the accountability system. Perhaps this 
will happen more and more as parliamentarians (and the 
press?) become aware of the information power of the 
program evaluation process. We certainly believe that 
this Committee was agreeably surprised by what we had 
to offer to their wide-ranging study. 


We found that the atmosphere of the session varied, at 
times very collegial and at times quite imposing, but 
always probing and with an underlying sense of the 
urgency of the problem at hand. Fortunately, we had 
taken a course on Media Relations earlier, and this served 
us well to beware of and handle the hidden agendas and 
the less than straightforward nature of some of the ques- 
tions. When questioning touched on more loaded policy 
issues, we stuck strictly to the facts or the findings, that 
is, what the clients and stakeholders had told us during 
the evaluation. 


In retrospect, this was an exacting challenge, one that 
suddenly cast our work into a limelight we were unused 
to, but one which we would definitely welcome again. 
Our advice to anyone facing this challenge: prepare care- 
fully, have a firm grasp of the information, and under- 
stand thoroughly its implications for policy-makers. # 


George M. Paterson 


Dr. Paterson is Director General with the Audit and 
Evaluation Branch at Agriculture Canada. 


Program Evaluation 


OCG Practicum Information Sessions 


The following is a calendar of events for the 1989-90 
Practicum Sessions: 


_ Cost-Effective Administrative Management 
and Control — _ 


This first session will comprise a description of the 

process of reforming administrative policy followed 

by the presentation of three audit case studies 

related to cost-effective administrative manage- 
. ment and control. 


October 23, 1989 (half-day session) 
Government Conference Centre 
Centennial Room 


| Demonstrating Cost-Effective Programs 


This session will discuss the development and use 
of meaningful cost-effectiveness indicators from 
ongoing information systems and program evalua- 
tions using several departmental examples from 
Part IIIs and Annual Management Reports. 


December 1, 1989 (half-day session) 
Government Conference Centre 
Centennial Room 


Improved Performance Through Better 
Financial Information 


Speakers will focus on several issues surrounding 
the increased emphasis on cost-effectiveness, 
revenue generation, productivity improvement, 
and cash management imposing special demands 
on both central and departmental systems for 
improved reporting. 


February 20, 1990 (half-day session) 
Government Conference Centre 
Sussex Room 


Exploring Program Delivery Options 


At this last Practicum session, discussion will focus 
on a variety of ways that can be used to explore 
alternatives to the present delivery approach, such 
as through program evaluation, Most Efficient 
Organization (M.E.O.) studies and implementing or 
augmenting user charges. 


April 4, 1990 (full-day session) 
Government Conference Centre 
Centennial Room 


For all the above sessions please confirm your 
attendance with Joan Cater at 957-7180. 


meno ieee | ene Ee eo 
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This Edition’s Main Theme: 
Human Resources in Program Evaluation 


The theme of this Newsletter is 
Human Resources, an ever-growing 
federal government concern of 
the 90s. 


Rob Jones (Office of the Comptroller 
General - OCG) discusses the contin- 
uing development of the program 
evaluation community by describing 
current initiatives in human resource 
management. He also presents an 
overview of past efforts in this area. 
A definition of ‘program evaluator’ in 
the federal evaluation community is 
set forth. Some of the more interest- 
ing characteristics of program evalu- 
ators currently working in federal 
departments and agencies are 
presented. 


Next, Guy Leclerc (OCG) provides an 
interesting viewpoint on evaluation as 
a profession and a career. He wel- 
comes a dialogue with readers on this 
matter. 


Norm Steinberg (Transport Canada) 
presents a proposal for the OCG and 
various organizations that are stake- 
holders in the community. 


Hervé Déry (OCG) reports on the 
one-day workshop which the 
Canadian Evaluation Society held last 
October. The theme ‘‘Profiling the 
Program Evaluators”’ links appropri- 
ately with the main thrust of this 
edition. # 


Paulette Somerville 
Editor 
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The Continued Development 
of the Program Evaluation 
Community 


During this last year, the Program Evaluation Branch 
(PEB) of the Office of the Comptroller General (OCG) has 
increased the emphasis and effort given to the continued 
development of the evaluation community. This article 
describes our current initiatives in the area of human 
resource management. It also presents some preliminary 
results and future challenges. 


Business and government are now paying increased 
attention to the people aspects of the working environ- 
ment. Enlightened organizations realize that people are 
a major resource, and they attempt to meet more than 
just the minimum or legislated needs of employees. Many 
have studied the relationship between the needs of the 
organization and those of the people who work in it. 
Some have argued, for example, that attempts to 
enhance productivity will succeed to the extent that 
employee needs and aspirations are met. 


For the federal Public Service to be an attractive, produc- 
tive work environment, leadership is necessary. For 
program evaluation, such leadership includes efforts to 
see that the function both is, and appears to be, a worth- 
while and rewarding area of employment. There is little 
doubt that the quality of evaluations depends greatly on 
the human resources available to carry out and report 
on the studies. Thus a legitimate concern of both depart- 
ments and agencies, and the OCG, is how to maintain 
a continuing adequate supply of the right type of people 
for the evaluation function. 


Many share responsibility for the management of human 
resources in the evaluation function. Each department 
and agency is responsible for ensuring that appropriate 
people work in its evaluation function. To this end, 


each undertakes activities to best meet its own needs! 
The Public Service Commission (PSC) has the general) 
responsibility for the people who make up the Canadian) 
Public Service. The Treasury Board, as the employer, has 
certain responsibilities. The policy centres in the centra 
agencies also play a role in terms of the people in thei 

functional communities, to varying degrees at various! 
times. Finally, each person in the evaluation commun 

ity is responsible for the ‘‘management’’ of his or he 

own career as an evaluator and as a public servant. 


In the OCG, the policy centres are accountable in specific 
ways for their respective functional communities (finan- 
cial administration, internal audit, program evaluation). 
Since its formation, PEB has undertaken various activi- 
ties to help central agencies and departments with the 
professional development of the program evaluation! 
community. The goal is a healthy, effective, professional, | 
government-wide program evaluation community. 
Beneficiaries of these activities range from deputy heads, 
who have the primary responsibility for their evaluation| 
function, through to evaluators themselves. 


Both the expectations of the federal evaluation function 
and of the people who make up the evaluation commu- 
nity have evolved since the promulgation of the Treasury 
Board policy in 1977. Because of this, in late 1988 PEB! 
reviewed its past and present efforts in community 
development. This review (see details in ‘‘Past Efforts) 
Reviewed’’) noted that certain of our activities, such as! 
the Professional Development program, have evolved | 
over time. It also identified several emerging issues. 
relating to career paths, staffing, etc., which provided 
opportunities for an enhanced PEB contribution. A 
major finding was the need for better information about | 
the community prior to an enhanced contribution to its! 
development. Evaluators themselves had, at times, 
mentioned these issues and the desire for more infor- 
mation about community trends. 


Current Initiatives 


For 1989-90, PEB planned two main initiatives in the 
area of community development. Developing and dis- 
tributing a profile of the program evaluation communi- 
ty is the first initiative. It will increase knowledge about | 
the evaluation community for all parties involved in some 

way with its ‘‘management’’. Providing, on an ongoing 

basis, more effective support to the selection of 

Directors and Heads of Evaluation is the second 

initiative. 


We began this work mindful of certain facts. First are 
the known characteristics of the community. Our com- 
munity is relatively small and contemplated involvement 
is scaled accordingly. Our work should reflect the various 
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needs and wishes of the people. It should also reflect the 
desired and anticipated characteristics of the evaluation 
function as it continues to evolve. It should build on links 
with evaluators through ongoing PEB activities. The 
work must be defensible: it must make sense to people 
in the community. 


The entire Public Service is undergoing profound change. 
There was a time when mobility was high, jobs were 
plenty and aspirations towards upward mobility were 
common and often realistic. Over the past several years, 

it has become apparent that career paths have changed. 

There are no longer as many opportunities for obtaining 
new experiences, skills, and challenges through vertical 
obility. These and other aspects of life in the Public 
rvice have attracted much recent attention. 


ae 

In its Annual Report for 1988, the PSC notes that ‘‘the 
size, composition, and age structure of the Canadian 
Public Service have changed significantly’’ (p. 13) in the 
recent past. The Public Service is an aging work force 
and will continue to be so for awhile. Plateauing, or 
career blockage (‘‘structural barriers to career advance- 
ment arising due to a combination of age imbalances and 
a static or contracting work force”) is seen as a source 
of problems of morale, motivation, retention and 
underemployment. The report presents various indica- 
tors of the extent of this problem and reviews efforts 
aimed at rejuvenation of the work force and increasing 
opportunities for movement. 


The first PEB initiative examines the extent to which the 
changing times are affecting the program evaluation 
community. The resulting Community Profile will 
contain analyses of qualifications and previous work 
experience, career paths (including information on 
mobility), and the impact of employment equity pro- 
‘grams. We will send this document to all evaluators. The 
profile will be used as the basis of a strategy to address 
any present and anticipated community needs. 


Since this project started, the Treasury Board Secretariat 
published a discussion paper, ‘‘Shaping the Future Public 
Service,’ as the first phase of the project Public Service 
Work Force 2000. As the results of this project may 
influence our working life, its progress merits consider- 
ation. ‘‘Both private and public sector employers recog- 
nize that their employees are the key to maintaining a 
competitive edge in the marketplace of the 1990s. In fact, 
continuing to attract and retain skilled, talented and 
motivated employees is one of their main challenges” 
(Preface). Three major challenges are foreseen: 


® “an increasing proportion of the Public Service work 
force may need ongoing training and education, or 
retraining; 
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e the Public Service will likely compete in an 
increasingly tight labour market for highly skilled 
people; and 


e forecast skill shortages would require measures to 
retain highly skilled public servants.’ 


The second PEB initiative is the provision of enhanced 
support to the staffing process, especially at the 
most senior levels. This will be a continuing initiative. 
As in the past, the objective is to influence the quality 
of selection decisions. Good people are important to the 
present and future success of the function. There are 
other objectives for this work: to maintain opportuni- 
ties for career advancement within the function and to 
increase movement within the community. 


For competitions held by the PSC for a Management 
Category vacancy, we cooperate in the identification of 
capable and potentially suitable candidates from within 
the community. We provide this information to the PSC. 
This input is additional to the mandatory search of the 
Management Resources Information System (MRIS) con- 
ducted by the PSC. Other parties provide lists of candi- 
dates as well. Through participation on selection boards, 
we contribute to the identification of people who would 
be suitable for the particular position. Such involvement 
is limited to participating in assessing qualifications. The 
hiring manager has the final responsibility. 


We have also increased our efforts to help departmen- 
tal evaluation managers hire new staff. Based on 
appropriate criteria, specified by the manager, such as 
experience and educational achievements, we can iden- 
tify potential candidates from within the community. 
These candidates can then be considered within the 
staffing process. 


Where Are We Now? 


1. PEB has now developed a confidential computerized 
database containing information about the people in 
the program evaluation community. 


The first question was the definition of the program 
evaluation community: who was included? Consistent 
with our view of functional leadership, the definition 
is reasonably inclusive (see ‘‘Program Evaluators 
Defined’ for details). In choosing this scope, it is fully 
recognized that not all activities are equally appropri- 
ate for all people in all organizations. As examples, 
our professional development program is available 
to all members of the community regardless of the 
organization in which they work. On the other hand, 
we can most effectively support senior level staffing 
in organizations whose employees work under the 
Public Service Employment Act. PEB has a long- 
standing relationship with people in some Crown 
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corporations even though they are not subject to the 
Treasury Board policy on evaluation. People in these 
organizations contributed to the early development 
of the evaluation function. 


Program Evaluators Defined 


Development of the database of program evalua- 
tors was based on a number of factors, including 
its potential uses, the nature of the community in 
terms of perceived trends, the relative availability 
of relevant information, and the costs associated 
with its maintenance and use. Many of these fac- 
tors seemed to suggest the need for a database of 
people who have experience in the function which 
can be used flexibly to meet changing times and 
needs. We defined the community to include three 
types of people: 


1. Active Members: All people who currently 
occupy a program evaluation position in an 
organization of the federal government. 


. Inactive Members with Recent Experience: 
All people, currently employed within an organi- 
zation of the federal government who, within 
the past five years, occupied a program evalua- 
tion position. , 


. Interested People with Relevant Education, 
Training or Experience: People who have 
been identified and screened on the basis of 
relevant education, training and experience by 
the Program Evaluation Branch, Office of the 
Comptroller General of Canada. : 


For these purposes, a program evaluation position 
is one in an organization of the federal government 
whose substantive nature involves the management 
and/or conduct of all or portions of evaluation 
studies or the management of evaluation functions 
in line with the ‘‘Working Standards for the 
Evaluation of Programs in Federal Departments 
and Agencies’’, Office of the Comptroller General 
of Canada, July 1989. 


The first two criteria provide a focus on people who — 
either are working or recently have worked in 
evaluation. It is this group of people who have built 
the evaluation function. The inclusion of the third. 
criterion will provide the flexibility to mainiain a 
list of potential new recruits. = 


The second issue was the identification of informa’ 
tion requirements and availability, in both the mid 
and longer-term. As one can imagine, informatio 
about people can range from basic experienc 
through to expectations and career wishes 
Environmental information can include, for example 
human resource profiles of departments and agencies 
profiles of director level positions, potential or actua 
vacancies, assignment possibilities, etc. As always, th 
main determinant of information requirements is th 
set of uses which are envisaged for the information 


Based primarily on the needs associated with th 
development of a community profile, the followin 
information is the minimum requirement: 


¢ CURRENT EVALUATION POSITION: Departmen 
or agency, occupational group and level of person 
and position occupied, date of appointment to the 
position, status of the appointment (indeterminate, 
indeterminate - acting, acting on the basis of an 
assignment or secondment, term, Co-op student). 
Students are eligible for inclusion as they are) 
potential candidates for entry-level positions; | 


+ EDUCATION/TRAINING: Academic qualifications 
and specialities; 


¢ PREVIOUS EMPLOYMENT EXPERIENCE: Depart- 
ment or agency, nature of work, occupational 
group and level, of previous positions; 


e along with BACKGROUND information such as. 
language group and gender. Such information is 
available in a well-prepared Curriculum Vitae. 


In all this work, we assigned a higher priority to active 
evaluators. This choice will ensure a strong base of 
information into the future. 


We have now captured information from a number 
of sources. À primary initial source was the PSC. With 
its cooperation, we obtained and now maintain 
certain information from the MRIS; specifically, the 
one-page staffing printouts and available curriculum 
vitae. We have supplemented this information from 
other sources which include our normal day-to-day 
activities and our professional development program. 


We have also developed a procedure for ensuring the 
accuracy and completeness of certain categories of 
this information. Every three months, we will ask 
directors and heads of evaluation to verify and 
update lists of active evaluators. The first such verifi- 
cation took place in early December; we have 
received responses from almost all organizations. 


Although a lower priority, we have also begun to 
capture information about what evalutors do after 
they aren’t evaluating. The results of this effort are 
essential to the study of career paths. 


. We have also conducted some preliminary analyses 


of the evaluation community. We presented an early 
version to directors and heads of evaluation in 
September. The information presented in ‘‘What Do 
We Look Like?’’ reflects more complete data. From 
many perspectives, we are a rather dynamic group! 


. Partially in response to indications of demand from 


the community, we developed a proposal for an 
assignments program in the fall of 1989. Members of 
the Advisory Committee on Evaluation Policy and 
directors and heads of evaluation discussed it on 
separate occasions. In early December, in response to 
specific requests, PEB provided all active evaluators 


What Do We 


yn January 31, 1990, some 352 ne were he 

officers in program evaluation. Here are some of 
he characteristics of these people who work in over 
0 organizations of the federal see 


: bout 41% are in positions in the AG aiuctrative 
_and Foreign Service Category (mostly in AS group); 


_ about 25% are in the Scientific and Professional 
Category (majority in ES group); about 23% arein __ 


the Management Category. In terms of classifica- 
_ tion groupings, about 14% of the positions are at 

the EX (or equivalent) level, 15% at the SM (or 
_ equivalent) level, 25% at SM-1 level, 21% at SM-2 
_ level, and 25% at lower levels. 


Over 85% of the group have, at least, a Bachelors 
_ degree; over 55% have at least a Masters degree. 
This information is not available for about 40% of 
the group. These figures are based on the — 
assumption that no meaningful reporting bias exists. 


Evaluators report a diverse set of skills and 
knowledge from a wide range of specialties as 
_ follows: Arts (13%), Business, Commerce and 
Accounting (14%), Economics (15%), Education 


(6%), Engineering (7%), Mathematics (7%), Physical 


Sciences (6%), Political Science (7%), Psychology 
(6%), Public Administration (8%), Sociology (7%) 
and other Social Sciences (4%). 


The group possesses a wide range of experience in 
evaluation. The average is just over 4 years in evalu- 
ation, but about 17% have more than 8 years while 
about 40% has less than 2 years. 
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with information about several opportunities for 
assignments. The future of this proposed program, 
specifically, its rationale, is now being re-examined. Avail- 
able information indicates that many evaluators have less 
than two years of experience in evaluation while others 
have assumed new duties within the past two years. 


Future Steps and Challenges 


The next major step will be the publication and 
distribution of a community profile. This document will 
form the basis for a strategy for community 
development, reflecting the expectations of the evalua- 
tion function for the future. If this work is to be 
successful, people in the community must be both aware 
and involved. Consultation is essential. As a key step in 
this process, we have outlined our activities in this series 
of articles. This practice will continue. We seek the view- 
points of evaluators. @ 


Look Like? : 


e Only 28% of the group has had more than 1 set of 
duties as an evaluator. On average, this group has 
about 7 years of evaluation experience. About 14% 

_ has assumed new duties in evaluation in the last 
2 years. 


_e Directors and Heads average about 4 years of 


experience in the function but 42% have less than 
2 years and 16% more than 8 years. Slightly less 
than 40% had previous experience in the evalua- 
tion function. 


e 30% of the group are women; 26% are franco- 
phones. For the sub-group of evaluators employed 
under the Public Service Employment Act (PSEA), 
the corresponding figures are 32% and 28%. The 
comparable figures for officer categories in the 
Public Service are 29.1% and 26.3% (c.f. 1988 
Annual Report, Public Service Commission of 
Canada, page 88. We have been informed by the 
PSC that the figures cited herein will be slightly 
higher for 1989.) 


Women and francophones are represented at all 
classification levels of the community. For example, 
of those in the Management Category (EX and SM) 
employed under the PSEA, about 23% are women 
and 29% are francophones. The comparable partic- 
ipation rates for the Management Category as a 
whole are 12.3% and 21.4% (cf. 1988 Annual 
Report of the Public Service Commission, page 90). 


About 29% of Directors and Heads are women; 
about 31% are francophones. The corresponding fig- 
ures for the PSEA sub-group'äre 33% and 37%. @ 
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Past Efforts Reviewed 


Late in 1988, we conducted a review of our involvement 
in the overall development of the program evaluation 
community. This review focused on the specific roles and 
responsibilities assigned to us in the Personnel Manage- 
ment Manual (1983, Chapter 25, currently being 
revised). These are presented here in green. The main 
findings, which formed the basis for current activities, 
follow. 


‘‘Maintaining aggregate information on the program 
evaluation community.’’ 


During the early development of the function, there was 
less apparent need for information collected on a sys- 
tematic basis. Information collected and shared infor- 
mally was adequate for.the majority of known needs. The 
(then) current Branch database of members of the evalu- 
ation community, which was developed as a means for 
recording attendance at professional development 
events, was used for many purposes, from production of 
mailing lists to a source of information on the communi- 
ty as a whole. We updated this information on an ad hoc 
basis, if and when resources were available. 


Recently, the growing need for more and better infor- 
mation became apparent. Issues identified included the 
potential uses of the information, minimum require- 
ments as well as the means for obtaining this informa- 
tion in a cost-effective manner, the advantages and 
disadvantages of defining the community in a narrow as 
opposed to a more global way (e.g., All program evalua- 
tors in the federal government or only certain ones. What 
about evaluators in the private sector?). 


‘‘Participating in the selection process of senior 
management personnel in the program evaluation 
function and upon request, in the selection of other 
departmental program evaluation staff.’ 


During the developmental phases of the function, PEB 
was quite involved in this area. During the period of 
consolidation, little turnover took place. In the previous 
12 months however, about one-third of the directors or 
heads of evaluation changed. Examination of these 
recent appointments revealed some interesting trends. 
For about one-half of the appointees, their new position 
was their first position in the evaluation function. Such 
a pattern of staffing activity, if it continued, would ulti- 
mately have implications for the career paths of officers 
in the lower levels. Increased flexibility of staffing 
through reassignment and redeployment was apparently 
also a factor: the majority of appointees did not change 
departments. 


Again, during the developmental phases, considerabl) 
effort was expended assisting departments or agencie 
with selection of their evaluation staff. PEB was fre 
quently asked to participate on boards. We are no longe 
the sole source of such assistance; individual depart 


ments can and do frequently help each other. 


‘‘Providing a series of professional developme 
events that respond to the changing needs of thi! 
program evaluation community.’’ 


PEB has always been involved in a substantial way b; 
providing a series of seminars or workshops and infor 
mation exchange sessions. These events range fron 
orientation workshops for new members of the commu 
nity to methodological workshops aimed at mort 
experienced members. Information exchange sessions art 
designed to facilitate the exchange of relevant practices 


We plan these events on the basis of perceptions o! 
needs, stated needs of individuals, feedback fron 
participants, knowledge of emerging events, etc. Front 
time to time, we survey the community. We also facili 
tate professional development through our support of th 
Canadian Evaluation Society. | 
Issues identified in our review included the deter 
mination of minimum information needs to promote the 
further professional development of the evaluation com: 
munity. Given that training needs have been identified 
what sources of this training can be identified (e.g. OCG, 
PSC, CCMD, universities, CES, etc.)? Given that alterna: 
tive sources of training can be identified, what is the 
proper role for PEB with respect to trans 


‘‘Providing departmental specific training sessions 
upon the request of the Director of Program 
Evaluation.” 


In the past, we had received many requests for sessions 
of this type. Sessions were designed to focus on the 
specific types of programs in the particular department. 
This approach was seen as a cost-effective means of 
training an entire evaluation staff in a rather rapid 
fashion. Requests of this type are now relatively rare, 
occurring most frequently in new organizations, with 
new directors, in small evaluation groups. 


‘‘Resourcing training sessions sponsored by the 
Public Service Commission.’ 


The extent of our involvement in this area has varied 
over the past few years. During the last six months 
however, we received requests for assistance with both 
the Senior Management Orientation Course and the 
Career Assignment Program. New resource personnel 
have become involved. This type of activity helps to 
promote the evaluation function. | 
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‘Providing advice on the organizational structure 
and classification of positions of program evaluation 
mits, upon the request of departments.” 


In the early days, PEB was often involved in this aspect 
of basic infrastructure development. Issues such as a 
‘better’? or the ‘‘best’’ location for the function in a 
particular department or agency were addressed. 
Generic job descriptions were prepared and distributed 
as aids. Complementing work with departments or agen- 
cies were occasional interventions with relevant central 
agencies on these matters. We were often asked by 
officials in the Treasury Board Secretariat to comment 
on TB submissions reflecting reorganizations and 
classifications. 


As the vast majority of departments or agencies have 
long since developed this aspect of their infrastructure, 
current work is limited in volume and tends to arise 
during times of reorganizations. The nature of the work 
remains essentially the same. 


The federal program evaluation community is relatively 
heterogeneous when compared with other staff commu- 
nities: positions held by program evaluators are located 
in a number of occupational categories and groups, 
reflecting the wide range of skills and abilities applica- 
ble in the function. This characteristic has implications 
for other aspects of community development. 


“Co-ordinating the maintenance of the prequalified 
staffing inventory for senior program evaluators and 
evaluation project managers (AS-6 to SM) and 
equivalent levels), in conjunction with the Public 
Service Commission.” 


With the assistance of the evaluation community and the 
PSC, PEB developed an inventory to meet a shortage of 
qualified evaluation personnel. At the time, the function 
was new; people with the right mix of skills and abili- 
ties were frequently reluctant to join. The inventory, 
which facilitated the entry of qualified people from out- 
side the federal government to the evaluation function, 
was relatively easy to access. 


We reconsidered the continued need for this inventory 
in 1986. The function had matured to the stage where 
internal candidates made themselves available for vari- 
ous competitions. Demand from directors and heads of 
evaluation was not high. Moreover, throughout the entire 
Public Service, increased emphasis was being placed on 
reassignment, retraining or redeployment of personnel 
affected by work force adjustments. The inventory 
received no further support and was dropped. 
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‘Representing the program evaluation community 
in matters relating to staffing.’ 


PEB never has been very active in these matters as selec- 
tion or classification standards unique to the evaluation 
community do not exist. Our involvement has related to 
the development and use of the prequalified inventory, 
input to a terminology review for the Management 
Resources Information System, etc. = 


Rob Jones 


Dr. Jones is Senior Policy Analyst with the Program 
Evaluation Branch, OCG. 


Evaluation as a Profession 
and a Career 


The topic ‘‘evaluation as a profession’ is properly the 
domain of evaluation societies. In this article I want to 
focus on evaluation as a career. However, it is impossible 
to discuss evaluation as a career, without first discussing 
evaluation as a profession. One may have a career 
without a profession, but seldom a profession without 
a career. My intention is to engage the federal evalua- 
tion community in further discussion on this important 
topic by demonstrating the diverse elements of the situ- 
ation as I see them. No attempt is made to resolve the 
issues. The demonstration is simply offered to add some 
discipline to the debate. 


Evaluation as a Profession 


Organized evaluation, as we know it today in the federal 
government and as practised by public sector institutions 
(both by insiders and by outsiders), is a very recent 
phenomenon - it is scarcely 10 years old - although it 
owes much of its intellectual capital to enterprises begun, 
in some cases, more than 50 years ago. (See Chelimsky, 
1985; and Rutman L. and Mayne J., 1985). 


Evaluation has rapidly become a serious intellectual 
undertaking. It has attracted a variety of Public Service 
employees who seek and thrive on the challenges of 
review work. It has a strong appeal for public-spirited 
people who see evaluation as a major instrument of 
public administration. In addition, a healthy 
entrepreneurial spirit motivates persons in the private 
sector who are attracted to evaluation. 
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At the same time, the range of activities carried out 
under the evaluation umbrella is very wide; the calibre 
of people involved in these activities varies as well. At 
one end of the spectrum, you find highly trained people 
who act in a professional manner. At the other end of 
the profession, as in many others, there are imitators and 
opportunists. They conduct reviews without any 
discipline to back them up, simply performing, labelling, 
and selling reviews according to requirements. The latter, 
I hasten to say, are the small minority. 


Few institutions of higher learning offer a full range of 
evaluation courses. Few people in Canada, as far as I 
know, graduate from such institutions specifically as 
program evaluators. Indeed, very few people in Ottawa 
doing evaluation work ever aspired, as students, to 
become evaluators. No single educational stream or, for 
that matter, no specific on-the-job experience seems to 
lead naturally to the practice of evaluation; evaluators 
come from various places, probably as it should be. (See 
Box ‘‘What Do We Look Like?’’) 


Evaluation has not yet coalesced into a full-fledged 
profession. In my view, a profession consists of a distinct 
body of learned people exercising a particular activity 
using agreed-upon procedures, discipline, and standards 
of performance. The following describes briefly four 
requirements, at a minimum, for evaluation to become 
a profession in the rigorous sense of the term. 


First, there has to be a professional association with 
demanding and well-defined membership criteria which 
would exclude the unqualified. 


Second, the professional association must have a suffi- 
ciently distinctive name and must be in a position, legal 
or moral, to confer titles aimed at asserting some meas- 
ure of technical exclusiveness. 


Third, the association must develop standards of perfor- 
mance for the conduct of professional activities and must 
possess a code of ethics to regulate its members’ 
behaviour and keep out the unscrupulous. 


Fourth and most importantly, the association must 
promote itself by asserting in a convincing way the 
rationale and social utility of the profession. It also has 
to ensure its enhancement and renewal by engaging in 
or supporting professional training. The association must 
maintain a visible programme of public activities aimed 
at bringing the members together and engaging them in 
professional discussion, and it must be seen publicly to 
do so. Examples of this requirement are regular profes- 
sional gatherings, publication of professional journals, 
etc. 


Variations on these themes are found in the literatur 
of the sociology of the professions (e.g., Slayton P. 

Trebilcock M. (eds.), 1978). According to René Larouch: 
(1987), individual members of a profession have acquire‘ 
certain visible characteristics, such as a high level o 
income, prestige or influence; a high level of educatio 

and a great deal of professional autonomy or discretio 

The members are seen to be clearly committed t 
the profession and to enjoy a certain monopoly i 
relevant professional activities. In addition, members 0 
the professional society maintain for new members ¢ 
process of socialization into the group. 


Associations of evaluators in Canada have several of the 
elements mentioned above. They have done excellent 
work and deserve support, but they are what they cal 
themselves, interest associations or societies, and not 
professional bodies in the strictest sense of the word 
They do have enough of the basic attributes to earn the | 
the respect of client-organizations looking for profes: 
sional advice and are attractive for people seeking tc 
enhance their careers. | 
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Careers in Organizations and | 
Personal Careers | 
Hugh Gunz (1989) has introduced some interesting 
distinctions to bear in mind when discussing the notion 
of a Career in a given endeavor. His starting point is the 
distinction between a career as seen by an organization 
and an employees career in the mind of that employee. 


I believe, as he does, that the main problem is to deter- 


mine where these two concepts intersect. 


For an organization, careers involve attitudes that 
“career people” bring into the workplace and guaran- 
tee strong commitment to the long-term mission of the 
organization and to its renewal. For the career person, 
on the other hand, choosing a career or deciding to 
pursue a Career in a given domain suggests the opportu- 
nity to make considered decisions concerning multiple 
alternatives, successive or concurrent, which the work- 
place offers. The big question is how to reconcile the 
institutional requirements with the career paths of 
individuals. 


In this context, the distinctions of DT. Hall (1976) are 
useful concerning the meaning of ‘‘career’’ as seen by 
individuals in organizations. First, career may be seen 
as advancement. In this model, career is seen as a verti- 
cal (upward) movement, as a result of successive appoint- 
ments. Second, career may be seen as the practice of a 
profession, which for the professional is seen as advance- 
ment within a particular domain. Third, career may be 
seen as a series of successive appointments or the 
assumption of different roles, not necessarily related but 
taken together over the long term adding up to what the 
individual calls a career, a ‘‘diverse career’ (when not 
a checkered one!). | 
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Evaluation as a Career 


In the above distinction, there is obviously room for the 
notion of a career as an evaluator. But where does the 
career evaluator fit in the federal government context? 
Evaluation is recognized as a formal review process and 
follows a disciplined approach, as described, for exam- 
ple, in the OCG Working Standards for the Evaluation 
of Programs in Federal Departments and Agencies. 
Practitioners or evaluators should have a mastery of the 
discipline. In principle, there is no limit to the need to 
develop and apply the best knowledge and techniques 
in support of evaluative efforts. 


Moreover, organizationally speaking, evaluation has 
been explicitly identified as an official corporate func- 
tion. Thus, it has a distinct strategic position. This healthy 
organizational disposition towards evaluation should be 
very attractive to career evaluators, in principle at least. 
Unfortunately, sometimes, by design or as a result of 
inappropriate position classification the head of evalua- 
tion reaches his or her hierarchical limit too early and 
at too low a level. In most departments, the head of 
evaluation is ranked among the lowest corporate 
managerial positions. As a result, the person who wishes 
to have a career in evaluation may not be able to find 
satisfaction if advancement is their chief criteria of 
success unless he or she is prepared for early 
“Dlateauing’’, to use a current expression. Of course 
nothing precludes an individual from considering a 
career strictly for advancement in spite of the above limi- 
tation. One can always use evaluation as a ‘‘career move” 
or stepping stone especially early in life by entering 
evaluation with the intention of leaving it later at the 
appropriate time. 


This being said, it should be noted that hierarchies as 
we have known them in the last decades have been 
undergoing changes, some of them resulting in the 
frequently expressed feeling of plateauing, justified or 
not. Hierarchy is not the power base it once was. 
Horizontal dealings have become just as important as 
vertical ones. Greater professional freedom in the work- 
place has extended the network of relationships of 
practitioners among themselves, among their peers, and 
in relation to their clients. Having equal access to infor- 
mation from the top down makes for fewer and smaller 
differences between managers and managed people. 
There are other changes worthy of mention, but the fact 
is that a career evaluator in a contemporary setting need 
not be overly worried by hierarchical constraints as much 
as he or she might have been in the past. The number 
of channels available to promote evaluative activities and 
increase influence today is greater than it has ever been. 
In this environment, I might add there is no substitute 
for competent work. 
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The following is my bias in this discussion. Given my par- 
ticular role, I should be more selfishly concerned with 
organizations than individuals. Clearly, I have to worry 
about the well-being and interests of the latter to ensure 
the health of the former. For the good of federal evalua- 
tion as a whole (the organization across government 
departments and agencies) I must ensure its renewal. 
This must be done with career people. But which ones? 
Those who view evaluation as a transitory profession? 
Or those who have embraced evaluation as a vocation? 
Indeed, do we have the right mix of career people? 


Your comments are solicited... 


Guy Leclerc 


Mr. Leclerc is Deputy Comptroller General of Canada 
responsible for the program evaluation function in the 
federal government. 
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A Challenge for the Future 


The OCG has played an important role in developing the 
human side of the evaluation function. The significance 
of the early responsibilities assumed by the OCG for 
human resources are probably underestimated by many 
of us, even though we are the principal beneficiaries of 
their initial foresight and activity. Today the evaluation 
function is operating in a relatively uninhibited environ- 
ment. The skills required to describe programs, carry out 
survey work, or conduct various forms of complex anal- 
ysis are available to most departmental functions. We are 
also fortunate to share a common language and set of 
values that have emerged over time. 


The community that we know now did not exist seven 
years ago. Back then a few people had a general feeling 
for what ‘‘program evaluation’ meant but there was no 
precision to concepts or language. With a commitment 
to training and professional development the OCG, on 
behalf of us all, defined the common ground, created 
the occasions for us to meet in various forums and gener- 
ally nurtured the community along. In certain depart- 
ments they were central to our success in obtaining 
reasonable staff numbers and classification levels. In 
other departments they were more of a life support 
system, key in breaking down or overcoming internal 
resistance to the function and at times breathing life into 
it in the form of person years as well as hands-on 
expertise. 


It can be said that the early period of community 
development is over. We are now a reasonably mature 
community. While it is true that the community 
continues to see new people moving in and some older 
members move out, we have arrived. At this stage, it is 
time to reaffirm our commitment to the people we work 
with and to plan for the next phase by determining what 
the evaluation community is moving toward in the next 
five years and by putting into place a human resource 
plan that anticipates and will accommodate the future 
needs of the community. 


The OCG should continue to play a central role. Rather 
than helping build new organizations as in the past, they 
should watch over the paring-down process that is 
occurring across the Public Service generally to ensure 
that the evaluation community contributes its fair share. 
At the same time the OCG needs to monitor the impact 
the changes will have on the morale of the community, 
and in the context of Public Service Work Force 2000, 
the OCG should be formally articulating the experience, 
values and vision of the evaluation community. 


In recognition of the OCG’s previous success let me 
conclude this article with a concrete proposal. A human 
resource plan for the federal evaluation community 
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covering the next five years should be completed withir 
four months by the OCG. In recognition of the fact that 
the federal evaluation community is comprised of th 
OCG, line departments and agencies and the Office o 
the Auditor General, as well as members of the Canadian 
Evaluation Society and members of private secto 
consulting companies, it is recommended that all these 
organizations be invited to participate. I would suggest 
that the process could be launched with a one-day’ 
symposium which would hopefully sort out the poten- 
tial roles and contributions of each of the participants. = 


Norm Steinberg 


Mr. Steinberg is Director of the Program Evaluation 
Division at Transport Canada 


“gm 
a | 
National Capital Chapter 


Fourth Annual One-Day Workshop 


Canadian 
Evaluation 
Society 


The National Capital Chapter of the Canadian Evalua- 
tion Society held its annual one-day workshop on 
Tuesday, October 24, 1989, at the Ottawa Congress 
Centre; its intriguing theme was ‘‘Profiling the Program 
Evaluator’’. 


Iris Bradley of Labour Canada, organizor of the work- 

shop, welcomed the participants. Ruth Hubbard, 

Deputy Minister of Revenue Canada, Customs and 

Excise, gave the keynote address. She characterized 

evaluators as advisors whose judgement is based on 

objective, reliable information and who have an impor- 

tant strategic impact on the decision-making process in 

her department. In Mrs. Hubbard’s opinion, the principal 
qualities demanded of an evaluator are sensitivity, 

understanding, perseverance, sound judgement and 

effective communication - the same qualities expected 
of a senior manager in policy-making or strategic - 
management. Viewed in this light, the evaluation func- 

tion allows those involved in it to develop qualities 
essential to these management functions. 


During the lunchtime information session, the president 
of the National Capital Chapter, William Baker of 
Revenue Canada, Taxation, first commented on the 
Chapter’s maturity and the professionalism of its 
members. He then introduced the guest speaker « 


art 
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the Honourable Jean-Luc Pépin, former Minister of 
Transport and External Affairs. Mr. Pépin said that, in 
his opinion, judgement is the chief quality of a program 
evaluator. By comparison, politicians mainly require 
energy and balance to be capable of reconciling con- 
tradictory qualities with which they must deal. In his 
speech he focused primarily on the British political 
tradition to which the Canadian accountability process 
belongs, comparing it to the process followed by the 
British system. This system is based on the principles of 
‘ministerial responsibility, merit, safeguards, neutrality, 
impartiality, anonymity, loyalty, detachment and finally, 
frugality. Mr. Pépin felt that these principles are being 
_ eroded, especially ministerial responsibility. In analysing 
these principles, he identified contradictions between 
the 1985 McGraw report and the Privy Council’s posi- 
“tion. Mr. Pépin felt that a position somewhere between 
‘ministerial and administrative responsibility must be 
“developed and that the CES might have a role to play 
in seeking this middle ground. He said that his many 
travels around the world had taught him that the main 
difference between industrialized and developing nations 
lies in the type of relationship that exists between 
politicians and public servants. 


Guy Leclerc, Deputy Comptroller General, ended the 
annual one-day workshop by summarizing its events and 
major thrusts. He briefly outlined the differences that 
he has identified between Canadian and American evalu- 
ators. In his view, the practice of evaluation in Canada 
is directed more toward strategic decision-making. 
Mr. Leclerc took the opportunity to stress the OCG’s 
interest in evaluator career development. He felt that 
‘it would be wise to achieve an equilibrium between 
individual and organizational needs. Supported by data 
from a survey conducted that morning among 
participants at the annual workshop, Mr. Leclerc main- 
tained that, although the profession was often seen as 
a springboard to senior management positions, the func- 
tion offered career prospects as well. Mr. Leclerc felt 
that, to strike a harmonious balance between these two 
career paths, evaluation managers should be classified 
upward to a level more comparable to that of managers 
of other departmental management functions. This 
solution is based on the results of a recent study by a 
department, which shows that consultant firms employ 
relatively more senior analysts than departments. The 
market thus recognizes the status of evaluators and, 
implicitly, the strategic importance of evaluation in the 
decision-making process. This direction should therefore 
be pursued. 


I would also like to acknowledge the contributions of the 
following federal public servants who participated in the 
simultaneous sessions: Alain Barbarie (EMR), 
Sheila Batchelor (RC-CE), Mike Binder (DOC), 
Terry Burbridge (CEDC), Ron Jackson (Lab), Guy 
Leclerc (OCG), Ruth Matte (OCG), Helen McDonald 
(DOC), David Moynagh (OCG), Shane O’Connor (CNR), 


Program Evaluation 


Joyce Potter (CMHC), Michael Stephens (TBS), Toru 
Uno (OCG) and Alan Winberg (TBS). a 


Hervé Déry 


Mr. Déry is Evaluation Analyst, Program Evaluation 
Branch, OCG. He is also Treasurer for the National 
Capital Chapter of the Canadian Evaluation Society. 


Merit Award Recipient 


On December 1, 1989, Mr. W. David Moynagh, Chief, 
Science, Economic and Regional Development with the 
Program Evaluation Branch (OCG), was presented with 
an OCG Merit Award for his performance at an unusu- 
ally high level over several years and for the successful 
completion of a number of tasks and projects in a manner 
beyond what was expected by management. 


Mr. Moynagh has applied intellectual abilities, a varied 
educational background and his personal dedication to 
his work, which he pursues with zeal and a sense of 
purpose. The program evaluation profession has greatly 
benefited from his incredible energy. He is the co-author 
of the Canadian Comprehensive Auditing Foundation’s 
major book entitled ‘‘Effectiveness - Reporting and 
Auditing in the Public Sector’’, one of the most widely 
distributed books in auditing in recent years. He has 
contributed substantially to the promotion of evaluative 
concerns in the strategic management of Science and 
Technology. He has played a dominant role in delineating 
program performance features of a particular IMAA TB 
- Memorandum of Understanding. Through his 
meticulous and dedicated work, he has gone a long way 
to converting program evaluation principles into practical 
applications. 


From left to right, W. David Moynagh (PEB) recipient of an OCG 
Merit Award presented by Andy Macdonald, Comptroller General. 
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Program Evaluation 


PEB’s PRACTICUM Session: 
Determining the Cost-Effectiveness 
of Programs 


For the ninth consecutive year, the Office of the 
Comptroller General is organizing several ‘‘Practicum’’ 
sessions. The main theme for this season is the exami- 
nation of the ways and means to increase program cost- 
effectiveness. The first session this year dealt with 
‘Cost-Effective Administrative Management and 
Control’. 


The Program Evaluation Branch (PEB) was responsible 
for preparing the second session, ‘Demonstrating Cost- 
Effective Programs’’, during which a number of 
speakers shared their views and experience. This ses- 
sion was held at the Conference Centre on December 
1, 1989. 


An audience of 120 people heard John Mayne, Director, 
Evaluation Policy (OCG), describe better cost- 
effectiveness in the federal government as ‘increasing 
the value received in relation to the resources used’’, 
and explain how cost-effectiveness should be measured 
and reported. George Willis, Director, Reporting Stan- 
dards and Practices (OCG), expanded on this theme, 
emphasizing the importance of how the information on 
program cost-effectiveness is to be reported and the 
need for a good strategy to this end. This could mean 
that it is necessary to consider a number of different 
clients (deputy ministers, managers, Parliament, etc.) 
to whom this information will be given. 


The first practical example was supplied by Revenue 
Canada-Customs and Excise. Three speakers discussed, 
from their respective points of view, the process that 
led to ‘‘Cost-Effectiveness in the Customs Postal 
Operations’. 


Earl Warren, Director General, Commercial Operations, 
outlined the main elements of the program (mandate, 
activities, resources, revenue generated, etc.). The 
transformation of the program, with the inherent cru- 
cial policy decisions, did not occur only in terms of its 
objectives but also involved the operational level, in- 
cluding employees and clients, explained Robert 
Struthers, Chief, Postal Operations Programs. 
Moreover, the criteria for assessing the achievement of 
the objectives were set to enable evaluation of the iden- 
tified results and therefore make the necessary cor- 
rections. 


Nicole Mendenhall, Director, Program Evaluation, 
discussed the role played by program evaluation in 
establishing the cost-effectiveness of postal operations. 
It could be a large one or a small one, depending on the 
point of view adopted. The evaluation function was 
the invisible mainspring of this analytical exercise. The 


cost-effectiveness indicators made it possible to show 
that the restructuring had positive results, even thou 9 
it was difficult to say with certainty that the program 
was cost-effective since it had been restructured and 
was, in fact, a new program. Each evaluation did | 
however, reveal ways to improve the program’s effec: 
tiveness. 
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The second example was also presented by three 
speakers. In this case, participants examined ‘‘Cost- 
Effectiveness in the Government Telecommunications 
Agency (GTA)’’. Roger Beauparlant, Director, Planning 
and Coordination of GTA gave an overview of the past. 
and present situation and resources (revenue, operat- 
ing costs and person-years) and described the perfor-' 
mance criteria used by GTA which included the 
principles of cost-effectiveness. 


The Agency was evaluated by the Program Evaluation! 
Branch of the Department of Communications as part | 
of a comprehensive review carried out by the Treasury | | 
Board of all GTA activities. 


According to Robert Letellier, Senior Program Evalu-! 
ation Manager, the Agency is cost-effective. However, | 
there are two lessons to learn from the cost- 
effectiveness analysis: (1) ‘‘It is more interesting and | | 
more useful to measure the degree or degrees of effec- | 
tiveness (effectiveness optima) of an organization such 
as the GTA than to measure only its overall effective 


ness. However, you have to be prepared to pay the price; | 


and (2) the choice of cost-effectiveness indicators must* 


be made carefully based on data that is easily accessi- 


ble at a reasonable cost.’ 


Another aspect of effectiveness evaluation was” 


presented by Helen McDonald also of the DOC Program 
Evaluation Branch. It consists of a series of performance 
indicators developed for the IMAA program which will 
make it possible, among other things, to measure the 
Agency’s effectiveness. These indicators will be part of 
the annual report to be submitted by DOC to the 
Treasury Board under the IMAA Agreement and will 
also be included in a special evaluation in 1992-1993. 
The indicators will be measured using previous evalua- 
tions and data collected and analysed by the program 
managers. In connection with the above evaluation, we 
already know that clients are generally very satisfied 
with the services provided and that the Agency is very 
cost-effective. The upcoming performance evaluations" 
will determine whether the GTA is evaluat Ose its 
programs properly. 


Lastly, Michael Stephens, Group Chief, Expenditure 
Analysis and Systems, Treasury Board Secretariat, listed 
the ‘‘Expectations for Cost-Effectiveness’’ of depart- 
mental programs. He mentioned that IMAA will help 
departments direct their efforts to provide better results 
with the resources available. Mr. Stephens also stressed 
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that good cost-effectiveness indicators are an essential 
link between results and resources as they establish the 
relationship between the output - the benefits and posi- 
tive impacts - and the input - namely the financial and 
human resources. Good cost-effectiveness can be 
achieved if the indicators are adapted to each depart- 

ment and they are sufficiently strategic to be useful to 
both the departmental managers and to managers of the 
central agencies. 


Practicum continues with ‘‘Improving Performance 
through Better Financial Information” and ‘‘Exploring 
Program Delivery Options” to complete the ninth 
season. = 


Ibolya Galambos 


Mrs. Galambos is a research analyst with the Program 
Evaluation Branch. 


News Briefs 


On November 10, 1989, Howard Fancy, Deputy Secre- 
tary, Budget Management Branch of the Treasury of 
New Zealand met with several officials of TBS, Finance 
and OCG. Mr. Fancy met with Guy Leclerc, Deputy 
Comptroller General, PEB and John Mayne, Director, 
Policy Evaluation to discuss the role of program evalu- 
ation in the Government of Canada and, in particular, 
in expenditure management. 


On November 16, 1989, Didier Maillard, Assistant 
Director, Sectorial Studies and Policy with the Depart- 
ment of Economics and Finance for the Government of 
France, met in Ottawa with representatives of the 
Departments of Finance, Transport and the OCG. 
John Mayne, Director, Evaluation Policy Division 
discussed various aspects of public policy evaluation 
with Mr. Maillard. 


Program Evaluation 


The Canadian 
Evaluation 

society 
CONFERENCE ‘90 


The Annual CES Conference will be held at the 
Hilton International Toronto, on May 14-16, 1990. 
The theme will be ‘‘New Perspectives on Evalua- 
tion for the 90s’’. Conference ’90 will bring 
together hundreds of evaluators and users of 
evaluation from the business, government, 
academic and voluntary sectors across Canada to 
share perspectives and the latest developments in 
the field, as well as to explore new ways of meeting 
the challenges ahead. 


For more information you may contact: 


Conference CES ’90, 

c/o Signature Programs Canada Inc., 
Conference Coordinators, 

37 Madison Avenue, Suite 300, 
Toronto, Ontario. M5R 282 

Tel.: (416) 924-1956 


Do you have an interesting article on program 
evaluation that we can publish? The Newsletter 
welcomes all contributions in either official 
language from the evaluation community. For 
further information, please call the Editor, 
Paulette Somerville at (613) 957-7167. 
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IGHLIGHTS: 


4 
# 


ew OCG Initiatives : 1 


ogress in Part Ill Evaluation 


potting 3 
novation in Evaluation À 
En of the Government's 
valuation Policy 7 
ist of Completed Evaluation 
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New OCG Initiatives 


I n this fiscal year, the Program 
Evaluation Branch of the OCG 
will be continuing a number of 
initiatives. Among them are the 
Treasury Board program evalua- 
tion policy review, further work 
on evaluating rationale/alterna- 
tives, documenting utilization of 
evaluation and our work in regu- 
lation and science and technology. 
A number of new initiatives will 
be undertaken. 


Evaluating Service 
to the Public 


Service to the public has long 
been a concern in government, 
and is likely to be more so in the 
90s. While federal evaluations 
have dealt with questions of the 
quality of service delivery, the 
extent and nature of the findings 
and impacts of evaluations in this 
area are not well documented. 
The PS2000 Task Force on 
Service to the Public has asked 
the OCG to review evaluations 
which have dealt with service 
levels and to provide input into a 
review of government studies 
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which have examined the quality 
of service to the public. In 
addition, we will be exploring 
methodologies which have been 
used and the kinds of service 
delivery alternatives which have 
been examined. 


During the upcoming year, the 
Branch expects to report on best 
practices, successes and the 
challenges of conducting eval- 
uations on the quality of service 
delivery. Please let us know of your 
best practices in this area. Contact 
Allison Fader at 957-7230. 


Grants and Contributions 


Terms and conditions of grants 
and contributions typically 
include an evaluation clause. The 
evaluations done as a result of 
such clauses range from project 
completion reports by the 
recipients to comprehensive 
evaluations by the corporate 
program evaluation group. But in 
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most cases, there is little government-wide 
knowledge about the evidence provided and its 
utilization. In the special case of ERDAs, we have a 
better base of information on the evaluations 
carried out. 


During the year, we will be examining the nature, 
extent and usefulness of evaluations done on 
federal grants and contributions. If you have any 
experience or information that might be of interest 
to us in this work, please contact Rob Jones at 
957-7229 or Toru Uno at 957-7185. 


Evaluating TB Policies 


As a result of the IMAA and PS2000 initiatives, all 
TB administrative policies are being reviewed with 
the intent of reducing the number of requirements 
to be followed and changing the focus of the policy 
from procedures to results. In planning present 
and future reviews of these policies, and in 
monitoring them, questions on their effectiveness 
and the cost in departments of implementing them, 
rather than questions of compliance with the 
policies, are in the forefront. 


The issues surrounding a review of a TB policy 
typically involve a large number of audit issues, 
but increasingly evaluation-type issues are also 
being posed. Internal audit groups are being 
challenged to move to a results focused audit and 
away from compliance audits. We have been 
reviewing the role departmental and central 
agency evaluation groups could play as these 
reviews of TB policies become more the norm. A 
draft paper has been prepared and distributed to 
departments for comment. Several departments 
have been involved with some of the review 
initiatives of the Administration Policy Branch of 
the Treasury Board Secretariat. During the year, a 
number of pilot cases will be monitored to better 
understand how evaluation can best contribute to 
the review of TB policies. 


Please contact John Mayne at 957-7177 for further 
information. 


A Focus on Evaluation Methods 


This year, we will develop a strategy for meeting 
the methodological challenges for evaluation in the 
90s. The branch will be assisted in this area by an 
advisory group drawn from the evaluation 
community. In our view, managers and other users, 
both internal and external, will have increasing 


need for pertinent, clear and convincing evide! 
on complex evaluation issues. Special attentp 
will be paid to methods used by department: 
generate and assess program delivery alternatie 
and to evaluate program rationale and cdi 
effectiveness. Professional development events ¢ 
publications focusing on methods will also! 
reviewed. In addition, a number of meth 
discussion papers will be published this year. 


Please contact Cameron McQueen at 957-7 ! 
for further information. 


Working Paper on Evaluation of Regulatory 
Compliance | 


This year, PEB is preparing a paper on evalua 
of regulatory compliance. The main objective 
this initiative is to provide a useful tool! 
evaluators who have to evaluate regulatd 
programs. The term compliance refers to il 
effective implementation of regulatory poli 
objectives. Evaluation of regulatory compliar! 
refers to the assessment of the extent to whi 
regulatory requirements are met by regulatees a 
the effectiveness of sanctions and procedures tl 
are employed to implement policy objectives. Ti 
question of compliance is unique to regulatd 
programs and important since a program W 
impact only to the extent that there is a certe 
level of compliance. It is also important | 
background information to evaluate the prograr 
achievement of its objectives and cost-effectivené 
of alternatives. Even though, for obvious reasoi 
the issue of regulatory compliance is not pertine 
when an expenditure program is evaluated, it m' 
become an issue if regulation represents | 
alternative to the evaluated expenditure progra’ 
In that context, the paper will be of interest to: 
the evaluation community. 


Please contact Hervé Déry at 957-7186 for furth 
information. M 
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ogress in Part Ill Evaluation 
feporting 


ince 1984, departmental Part IIIs of the 
stimates were to include information on program 
esults and, where appropriate, action plans 
erived from program evaluations. This does not 
jean that every finding from every evaluation 
ught to have been reported. Departments were to 
aclude only information that was significant to 
arliamentarians. 


ast fall, the OCG undertook a study of the 
art III evaluation reporting history of 
7 departments. Not every department would have 
ad significant evaluations each year but a positive 
rend was observed. Only 12 departments 
sentioned evaluation in their 1985/86 Part IIIs. By 
989/90, 32 of the 37 mentioned evaluation. 


n the five years covered by the study, the 
7 departments completed 520 evaluations among 
hem. Of these, 316 were mentioned in Part IIIs. 
‘ome of these studies may have lacked signifi- 
ance; some others perhaps ought to have been 
eported. And, of course, there may have been 
ome studies reported which were not all that sig- 
ificant! The OCG did not explore the reasons for 
ot reporting evaluations. However, the percentage 
f completed evaluations which did get reported 
‘ose from only 24% in 1985/86 to 81% in 1989/90. 
This improvement must be some combination of 
wo possible and welcome tendencies — more con- 
cientious reporting and/or fewer evaluations of 
ow interest to parliamentarians. 


More departments are reporting more evaluations. 
But what was the nature of the reporting? Some 
-eferences to evaluation indicated only that a study 
was pending or in progress. This category was 
xcluded from further analysis because subsequent 
Part IIIs would provide opportunities for fuller 
reporting. For completed studies which were 
mentioned in Part IIIs, five types of reporting were 
documented: 


* completion of the study merely noted; 
* findings reported; 

* study conclusions given; 

* recommendations reported; and 


* actions based on the evaluation discussed. 


Note that these categories do not constitute a clear 
hierarchy. There are several problems. First, only 
significant findings need be reported, although 
reporting without any study results doesn’t seem 
too meaningful. Second, conclusions, recommenda- 
tions and action plans are to be included only when 
appropriate and significant. Also, departmental 
actions may lag study completion by months or 
years. Thus, a study reported as complete in 1988 
or 1989 might not have had its full impact yet and 
reporting may not be complete. Some departments 
may be reluctant to report the same evaluation in 
two or more Part IIIs thereby inhibiting the 
reporting of delayed reactions. However, many 
departments do mention evaluations in more than 
one year’s Estimates. Only the fullest level of 
reporting observed for each study was used for this 
analysis to avoid double-counting. 


Of the 316 completed and reported evaluations, 44 
or 14% were reported along with departmental 
action plans. Recommendations but no actions 
were reported for 68 (22%). Conclusions (but no 
actions or recommendations) were reported for 
another 68. Findings only accounted for 98 (31%) 
and 38 (12%) were mentioned only as completed. 
Among these categories, no patterns are apparent 
over the five-year period the study looked at. 


os Part Ill Reporting : 
of Program Evaluation 


ee | Number of Studies Reported us 
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Clearly, many departments are using evaluations 
to report on program performance in Part III of the 
Estimates. However, it is not clear that Parliament 
is hearing about all potentially interesting studies. 
The level of the information reported varies 
widely. @ 


John Batchelor 


John Batchelor is Acting Chief, Regulation, Taxation 
and Financial Institutions Sector, PEB /OCG. 


Innovation in 
Evaluation 


Finding new and creative 
ways to _ introduce 
program evaluation is 
paying off for the National 
Energy Board (NEB). 


Given the small size of the Board and hence 
limited resources, the fact that the Board’s 
activities are already reviewed by a public hearing 
process on an on-going basis, and the status of the 
Board as an independent quasi-judicial tribunal 
with legal status as a court of record, management 
had formed the opinion that program evaluation 
was inappropriate and redundant. Our challenge 
was to clearly establish the usefulness of a 
program evaluation function. A little creativity was 
in order. Alternative ways to perform program 
evaluations had to be found to convert program 
evaluation principles into practical applications to 
firmly establish this new function at the Board. 


The experience which we are about to share with 
you is the result of an extensive internal 
consultation process and calls for the involvement 
of the line manager in an innovative way of doing 
program evaluation. This strategic approach, a 
product of the peculiar conditions and resources 
available to us at the time, introduced program 
evaluation as a useful management tool to a 
somewhat reluctant management. 


The NEB is a relatively small regulatory agency 
and is composed of nine permanent and two 
temporary members who are supported in their 
work by a staff of about 330. Under the terms of 
the National Energy Board Act, the Board is 
accountable to the people of Canada through 
Parliament, and to Parliament through the 


Minister of Energy, Mines and Resources.f 
Board regulates gas and oil pipelines, internat; 
power lines, pipeline tolls and traffic, prove 


activities: Corporate Planning (OPF, IMAA-M 
etc.), Internal Audit, and Program Evaluation. 


comprehensive plan and undertook its fr 
evaluation in 1989. During the same period, ]] 
management was of the opinion tha 
fundamental and extensive review of the En! 
Supply Branch was necessary for two pring 
reasons: 


(1) the way in which the Board fulfills its mané 
had changed somewhat in recent years; | 


(2) there was concern about the extent to wi 
the Board’s activities in this area mi 
duplicate those of other agencies and a de 
to maximize inter-agency cooperation 
coordination in supply analysis. 


In looking at options for a fundamental reviei 
the Energy Supply Branch, three alternatives v: 
identified by management. The first option, to! 
resources internal to the Planning and Rev 
Branch, was viewed as impossible because of | 
of resources and of combined techni 
managerial, and industry expertise. The sec 
option, to contract out, was also rejected, mai 
because of the difficulties in linking an evalua’ 
with the decision-making process to the ext 
required in this branch and also because | 
experience gained during the conduct of the rev 
would be lost to the Board. The third opt 
favoured a study internal to NEB with stri 
involvement of a new director. The advantages 
this option are significant. It is relatively eas 

link the evaluation with the decision-mak 
process when the line manager is committe 

implementing the recommendations that al 
developed himself. Active involvement on the p 
of evaluation staff means technical expertise | 
this case in the area of oil and gas supply & 
demand conditions) is not lost at the end of: 
review. And of course, with the line manager 
staff, no contract dollars are expended. Of { 
potential disadvantages, the first is a perceptior 
lack of objectivity. The line manager still has ot] 
day-to- day responsibilities and hence less cont 
vis-à-vis evaluation deadlines. Also, it is difficult 
find a line manager whose willingness to emb: 
on such a project is seasoned with an analyti 
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ind suitable to the evaluation framework. 
inally, a Director of Evaluation cannot always 
xpect that the liaison officer from the Office of the 
omptroller General (OCG) can devote the level of 
‘me and energy required for this type of project. 
ption three was retained as the best alternative 
ecause it was thought by senior management that 
his fundamental review could best be done by a 
erson with training and experience in petroleum 
eology and/or reservoir engineering, who was 
amiliar with the industry and the role of the 
oard, but who would be new to the Board and 
herefore able to supply a fresh look at the role and 
he duties of the branch. 


he line manager performing the evaluation is 
rom outside of the federal Public Service, assigned 
o NEB under the Executive Exchange Program for 
period of two years. Upon termination of his 
ssignment, the director will return to private 
ndustry. As part of his assignment, the new 
lirector received a clear mandate to review, in 
ollaboration with the Planning and Review 
ranch, the mandate of the Energy Supply Branch 
n light of the Board’s current and prospective role 
ind to prepare options on what that mandate 
hould be. It was also understood that 
ollaboration with the Planning and Review 
ranch would result in the production of two 
eports: the Evaluation Assessment Report and the 
ivaluation Study Report, thereby meeting the 
standards of the OCG. 


n order to deal with the potential disadvantages of 
his approach, a strong advisory committee was 
reated. The advisory committee is composed of 
nembers from the Energy Regulation Directorate, 
Planning and Review Branch, and the OCG. The 
committee has the responsibility to review all 
>hases of the study/evaluation and to assist and 
advise on questionnaire development, study 
lesign, data analysis, development of recommenda- 
ions, reporting, etc. From the outset there was 
also a strong commitment to involve the Board’s 
new senior evaluator, both to ensure the integrity 
of the evaluation and to gain exposure to the 
energy regulatory environment and energy 
industry, as well as to involve the OCG to ensure 
objectivity and quality. 


The Evaluation Assessment Report was presented 
to the Internal Audit and Program Review 
Committee in August 1989. The recommended 
review approach, the evaluationstudy organization, 
and the schedule, as well as a list of client groups 
to be interviewed, were reviewed and approved by 
the committee. 


The purpose of the evaluation study was threefold: 


(1) to question the rationale of the Energy Supply 
Branch program; 


(2)to provide information regarding the 
performance and results of the program and to 
provide some measure of its cost-effectiveness; 
and 


(3) to assess opportunities to improve the 
functioning of the program, given that a 
continued need was established. 


The evaluation study began in the fall of 1989. The 
Principles for the Evaluation of Programs and the 
Guide on the Program Evaluation Function were 
sent to the line manager as background 
information. Areas of importance were highlighted 
and sample evaluation assessments were provided 
by the senior evaluator. These documents have 
proven to be useful reference material as the line 
manager worked through the review. Interviews 
were conducted with Energy Supply Branch staff, 
NEB personnel outside the branch, and with 
representatives of the Board’s various client 
groups, which included: producers, purchasers, 
consumer groups, other regulatory bodies, federal 
and provincial governments, transporters, 
aggregators, consultants, and academics — 
virtually a census of all possible opinions. The 
interviews were generally conducted by the 
Director of the Energy Supply Branch assisted by 
the senior evaluator of the Program and Review 
Branch. In conducting the Energy Supply Review, 
particular attention was given by the line manager 
to examining the extent to which it is possible and 
desirable to rationalize the Board’s activities in 
this area by greater cooperation with and use of 
data and analysis of other organizations. 


At the time of writing this article, the final 
evaluation report is being prepared by the line 
manager. The advisory committee will be meeting 
to discuss the findings and recommendations and 
the study will then be submitted to the Internal 
Audit and Program Review Committee for final 
approval. Decisions regarding the implementation 
of recommendations outlined in the report will be 
made by the Chairman and senior management 
within the Board. 


In conclusion, both the OCG and the NEB have 
found the experience positive. While this approach 
is not always possible nor desirable, it may 
constitute a viable alternative under certain 
conditions. It is possible to use this approach to 
program evaluation only when a strategy is 
developed to deal with the disadvantages, 


ee 
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especially the problem of objectivity. The major 
benefits of having used the line manager as the 
program evaluator have been tremendous with 
regard to cost, excellent integration with 
management insofar as the probability of use of the 
evaluation conclusions and recommendations, 
education of Board personnel about the values of 
evaluation, and technical experience gained — all 
aspects especially important for a new evaluation 
function. In this particular approach, as with any 
strategic effort, it was important to be aware of 
the pitfalls, but it was also possible to avoid them. 
What was needed was a line manager with 
openness, good analytical skills, willingness to 
participate, and commitment; strong involvement 
of the OCG in order to counterweight the possible 
lack of objectivity of line managers; functional 
authority of the Director of the Program 
Evaluation Branch and involvement of a senior 
evaluator; and clear deadlines and delineation of 
roles and responsibilities of each participant. We 
were fortunate to have all these safeguards in 
place. With this experience the NEB’s Planning 
and Program Review Group is continuing to strive 


Manager’s Comments 


While program evaluation has provided a useful 
framework in which to conduct the review of the 
National Energy Board’s (NEB) Energy Supply 
Branch, more important to the study’s success has 


been the acceptance by senior NEB management of 


the merits of periodically reviewing the Board’s 
activities to ensure that they are relevant and are 
producing the desired results. 


In my view, it is important to recognize that many 


line managers will also realize the value of 


conducting evaluations. While the objectivity of the 
line manager directly involved in an evaluation study 
may be of concern, managers themselves can often be 
expected to take steps to ensure the integrity of the 
study. This concern can also be mitigated by the 
formation of advisory committees or by conducting 
the study when the manager is relatively new to the 
position. 


The process which has been established for 
conducting the evaluation of the Energy Supply 
Branch has proven to be successful to this point in 
the study. In particular, my direct involvement has 
helped to define the issues to be addressed in the 
study and enhanced the information obtained 


Although future NEB evaluations are more lil 
to follow the classical norm, it is necessary for u 
meet particular management needs as t) 
present themselves. Our responses will contin 
be innovative and strategic. 


This experience has demonstrated to all par} 
involved that existing evaluation techniques a 
a considerable degree of flexibility and yet prod 
credible results if potential pitfalls are addres 
up front. Be as flexible and strategic as | 
situation requires, but think it through first. Thi 
after all the spirit of IMAA — look to your results| 


Hervé Déry Claire M.M. Scott | 
Evaluation Analyst Director | 
Program Evaluation Planning and Revi 

Branch Branch | 


Office of the Comptroller National Energy 
General Board 


4 
q 
through interviews with various client groups. In thi 
regard, it is important to have an individual involve 
in the interviews who is familiar with the technic 
subject area. Also, the interviews themselve 
provided a useful opportunity to contact variou 
clients. This will be useful on an ongoing basis 
although many of these contacts would also likel 
have been made in a less structured manner in th 
absence of the evaluation study. The involvement o 
the line manager should also help in implementin; 
the recommendations arising from the study. 


The principal disadvantage from the line manager’ 
perspective may well be the substantial time 
commitment necessitated by direct participation it 
the evaluation process, in addition to other ongoing 
responsibilities. However, in view of the potentia 
significance of the evaluation results, it is importan 
to ensure that the study be given an appear leve 
of priority. @ 


D.R. Collyer, 
Director, 
Energy Supply Branch 


leview of the Government's 


= 


-valuation Policy 


ast summer, the Comptroller General launched a 
eview of the (1977) Treasury Board policy on 
rogram evaluation, using the 1989 OCG Working 
jtandards for the Evaluation of Programs in 
'ederal Departments and Agencies as a framework. 
licy provisions have been examined in the 
ontext of government-wide management processes 
ch as the IMAA initiative and PS2000 and in the 
4 of managers’ and evaluators’ experience with 
aluation over the past decade. 


nput has been received from a variety of users of 
valuation, including a number of DMs and ADMs 
n departments, officials in central agencies and 
he Office of the Auditor General, and from the 
nembers (evaluation directors) of the OCG 
a Committee on Evaluation Policy. 


the review has been part of the general Treasury 
30ard effort to better target its policies. The review 
las also supported the PS2000 focus on: a 
wofessional Public Service dedicated to service to 
he public and valued for its important 
sontribution; innovative approaches to program 
a and enhanced accountability for program 

rformance and less emphasis on compliance with 
rocedures. 


Recently, heads of evaluation have commented on a 
draft of the revised policy which reflects the 
consultations to date. The draft distinguishes 
etween the mandatory elements of the policy (the 
musts”) and the “shoulds”, guidelines for 
suggested practices which have been developed by 
he evaluation community over the past decade. 


The draft portrays evaluation as a key element of 
both strategic management and accountability 
reporting. It outlines the responsibilities of deputy 
ministers and of central agencies for evaluation. 


Reasonable departmental evaluation coverage over 
time is a feature of the draft policy, but evaluations 
are expected to be done only when they would be 
material, timely, and cost-effective. The proposed 
policy reiterates the need for credibility in 
departmental evaluation, both with line managers 
and with persons outside the concerned 
department. 


The draft policy gives deputy ministers more 
flexibility in making arrangements for reviewing 
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departmental programs. At the same time, central 
requirements for departmental evaluation have 
been streamlined. Provision has also been made for 
a lead by the central agencies in the evaluation of 
complex programs of special interest to Treasury 
Board or other Cabinet Committees. 


In addition to explaining more fully the intent of 
the requirements of the policy, the draft guidelines 
discuss the evaluation community’s experience 
with such matters as the positioning of the 
corporate evaluation group in departments, the 
role of line managers in evaluations, relationships 
with internal audit, the advisory roles that many 
evaluation groups play which go beyond their core 
responsibilities, and public access to evaluations. 
These guidelines are intended to be read in 
conjunction with the 1981 Guide and Principles, 
the Working Standards and other OCG evaluation 
publications. 


Our expectation is that the policy and guidelines 
which emerge from the review process will serve to 
meet the evaluation challenges of the 90s and 
beyond. @ 


Cameron McQueen 


Cameron McQueen is Senior Policy Analyst with 
the Program Evaluation Branch, OCG. 


“The Canadian Evaluation Society 
National Capital Chapter 


Annual Workshop ~ 


“Evaluation in the ’90s: Evolving with the times” | 


Ottawa Congress Centre 
November 14, 1990 
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Federal Government Evaluation Studies Completed 


Last year, we published a list of evaluation studies 
in the Newsletter. As an update, we now present a 
list of studies completed since that time; that is, by 
March 31, 1990 and approved by May 15, 1990. The 
majority of these studies were completed during the 
last two fiscal years. Studies are listed by 
department/agency; if the title appears in both 


as of May 15, 1990 


French and English, versions exist in both official 


languages. 


Agriculture Canada 


Canadian Agriculture Market 
Development Initiative (CAMDI) and 
Production Development Assistance 
Initiative 


Canadian International Grains Institute: 
Federal Funding 


Economic Analysis of the Returns to the 
Seed Assurance Program 


Economic Analysis of the Returns to the 
Seed Potato Program 


Energy and Engineering Research 


Farm Adjustment Initiatives: Canadian 
Rural Transition Program, Farm Debt 
Review Board and Fund 


Federal Contributions: Protein, Oil, and 
Starch Pilot Plant Corporation 


Inventory of Canadian Agri-Food 
Research 


Research Branch Involvement in Official 
Development Assistance 


Special Canadian Grains Program (1987) 
South Saskatchewan River Project 


Atomic Energy Control Board 
Research Program 
Safeguards Support 


Canadian International Development 
Agency 
Evaluation corporative du secteur 
«Foréts» /// Forestry Sector 


Appui institutionnel à l’éducation post- 
secondaire /// Institutional Support for 
Post-Secondary Education (Executive 
Summary) 


Canada Mortgage and Housing 
Corporation 


Public Housing Program /// Programme 
de logement public 


Canadian Radio-Television and 
Telecommunications Commission 


Regulation of Radio 


Communications Canada 


Exhibit Transportation Service /// Service 
. de transport des expositions (sommaire 
exécutif) 


EXPO 86 /// EXPO 86 (sommaire 
exécutif) 


Fathers of Confederation Building Trust 
/// Fiducie de l’édifice des Pères de la 
Confédération (sommaire exécutif) 


Government Telecommunications Agency 
//| Agence de télécommunications 
gouvernementales (sommaire exécutif) 


Human Resources Management /// 
Fonction de gestion des ressources 
humaines 


Manitoba and Canada Economic and 
Regional Development Sub-Agreement 
on Communications and Cultural 
Enterprises /// Entente auxiliaire 
Canada- Manitoba de développement 
économique et régional sur le 
développement des entreprises des 
communications et de la culture 


Movable Cultural Property /// 
Programme de biens culturels mobiliers 
(sommaire exécutif) 


National Film Board Review (jointly with 
National Film Board /// Etude sur 
l'Office national du film (conjointement 
avec l’Office national du film) 


Tourism, Cultural and Multicultural 
Projects: 1986 /// Projet Tourisme, 
Culture, et Multiculturalisme: 1986 
(sommaire exécutif) 


Quebec and Canada Economic and 
Regional Development Subsidiary 
Agreement on Communications 
Enterprises Development /// Entente 


Correctional Service Canada 


The studies listed herein appeared on either of a 
two lists that were sent to the Public Accoui 
Committee by the Comptroller General of Canadai 
June of 1989 and 1990. As is normally the cae 
access to a report is subject to prevailing legislati 
and requests should be made through the originati 
department/agency. 


auxiliaire Canada-Québec de 
développement économique et régional 
sur le développement des entreprises q 
communication 


Consumer and Corporate Affairs 
Canada 


Evaluation des bulletins des corporatio} 
et de l’insolvabilité /// Corporations and 
Insolvency Bulletins (Executive 
Summary) 


| 
Food Sector Regulation Under cu 
Products 


Material Handling and Distribution | 
Section | 


Automatic Day Parole Review Provisior 
of Bill C-67 (jointly with National 
Parole Board) /// Dispositions du proje 
de loi C-67 relatives à l’examen 
automatique en vue de la semi-liberté | 
(conjointement avec la Commission 
nationale des libérations | 
conditionelles) | 


| 
Industries Inmate Pay Incentives | 


Private Family Visiting Program /// | 
Programme de visites familiales privées 


Summary of Individual Substance Abust 
Evaluations /// Résumé des évaluations 
individuelles du programme de 
toxicomanie 


Employment and Immigration Canada 


Computer Assisted Immigration 
Processing System /// Système 
informatisé de traitement des cas 
d'immigration 
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Inland Enforcement and Control /// 
Evaluation des activités de contréle au 
Canada et d’exécution de la loi 


Job Development /// Evaluation du 
programme développement de l'emploi 


Job Entry /// Evaluation du programme 
. d'intégration professionelle 


Local Economic Data Systems /// 
Evaluation des systémes de données 
économiques locales 


National Employment Service /// 
Evaluation du service national de 
_ placement (Rapport général) 


The Innovations and Entrepreneurial 
Management (TIEM) Corporation /// 


Evaluation du projet TIEM (Projet d’aide 


à la gestion d’entreprise) 


Unemployment Insurance Controls /// 
Contrôle d’assurance-chômage 


Unemployment Insurance Sickness 
Benefits /// Prestations de maladie 
d’assurance-chémage 


Energy, Mines and Resources Canada 
Coal Utilization Program 


-ENERDEMO (Energy Demonstration) 
Program: National Conservation and 
Alternate Energy Initiatives 


Explosives Act Administration 


Frontier Geoscience 


Environment Canada 
Climate Services and Research 


Commercial and Toxic Chemicals 


Industrial Effluent and Emission 
Controls 


Moose Jaw Water Filtration System 


Ocean Dumping and Marine Program 


External Affairs and International 
Trade Canada 


Asia-Pacific Foundation of Canada /// 
Fondation Asie Pacifique du Canada 


Consular Program 
Fairs and Missions 


Federal-Provincial Pilot Project on 
Co-location of Trade Representatives 
Abroad /// Evaluation du projet pilote 
fédéral-provincial de copropriété/ 
colocation pour la représentation 
commerciale à l’étranger 


Most Efficient Organization Studies 
(separate reports): Posting Services 


Provision of Accommodations 
Technology Inflow 
Trade Communications Service 


World Information Network (WIN 
Exports) 


Fisheries and Oceans 


Canadian Hydrographic Service 
Fishery Officer Career Program 


Great Lakes Fisheries Commission 


Health and Welfare Canada 


Drug Safety, Quality and Efficacy 


Health Promotion /// Étude d'évaluation 
de promotion de la santé 


New Horizons /// Étude d'évaluation du 
programme Nouveaux horizons 
(sommaire exécutif) 


Sustaining Grants for National 
Voluntary Health and Social Service 
Organizations 


Indian and Northern Affairs Canada 


Alternative Funding Arrangements /// 
Modes optionnels de financement 


Anishinaabe Child and Family Service 
Worker Training /// Programme de 
formation pour les services à l'enfant et 
la famille Anishinaabe 


Canada Yukon Economic Development 
Agreement 


Contributions to Inuit Cultural 
Organizations to Promote Development 
of Inuit Culture and Language /// 
Contribution financiére aux centres 
culturels Inuits pour la promotion du 
développement de la culture et de la 
langue Inuit (sommaire exécutif) 


Cultural Education Centres /// 
Programme des centres culturels et 
éducatifs (sommaire exécutif) 


Evaluating INAC’s Child Welfare 
Program /// Programme de bien-étre 
destiné aux enfants indiens (sommaire 
exécutif) 


Iliqvik Corporation: Training and 
Employment /// Activités de formation et 
d'emploi d’Iliqvik (sommaire exécutif) 


Indian Commission of Ontario 


Indian Forestry Resources ///Ressources 
forestières sur les terres indiennes 
(sommaire exécutif) 


Program Evaluation 


N.W.T. Land Use Planning /// 
Planification d’aménagement du 
territoire - T.N.-O. (sommaire exécutif) 


Post-Secondary Education Assistance 
Program /// Programme d’aide a 
l'éducation post-secondaire (sommaire 
exécutif) 


Systematic Approach for Evaluating 
Economic Development Programs /// Une 
approche systématique pour |’évaluation 
des programmes de développement 
économique (sommaire exécutif) 


Test Case Funding Program /// 
Programme de financement des causes 
types (sommaire exécutif) 


Youth Entrepreneurship Program /// 
Programme des jeunes entrepreneurs 
indiens (sommaire exécutif) 


Yukon Northern Land Use Planning /// 
Planification d’aménagement du 
territoire - Yukon (sommaire exécutif) 


Industry, Science and Technology 
Canada 


Business Opportunities Sourcing System 
Canada Awards for Business Excellence 


Canadian Institute for Advanced 
Research 


Import Intelligence Service 


Matching Grants Policy for Federal 
Funding for University Research through 
the Granting Councils (NSERC, SSHRC, 
MRC) 


Special Agriculture and Rural 
Development Agreements 


Justice Canada 


Legal Aid to Saskatchewan /// Aide 
juridique au Saskatchewan 


Labour Canada 


Equal Pay /// Evaluation du programme 
d’équité salariale (sommaire exécutif) 


Government Employees Compensation 
Act 


Women’s Bureau /// Évaluation du 
Bureau de la main-d’oeuvre féminine 
(rapport de gestion) 


Medical Research Council of Canada 


Training of Biomedical Scientists (Ph.D) 


National Defence 
Management of Information Technologies 


Centre, Foreign Language Training, 
Travel and Relocation, Dedicated Drivers 
(Chauffeurs), Word Processing, By-Hand 
Package Delivery, Library, Trade Fairs 
and Missions, Departmental Travel 


Indian Oil and Gas 


Northern Native Fishing Corporation /// 
Northern Native Fishing Corporation 
(sommaire exécutif) 


Training and Development of Civilian 
Personnel (Part 1) 


ee ee ee 
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National Film Board 


National Film Board Review (jointly with 
Communications Canada) /// Etude 


Excise Duty Function /// La fonction des 
droits d’accise 


sur l’Office national du film des recettes à l’Accise 
(conjointement avec Communications 
Canada) Postal Operations /// Etude d’évaluation 
sur la composante opérations postales 
‘ : Verifications and Enforcement /// La 
b 
Nana bre on Careds composante vérification et exécution de 
Public Services /// Service au public l’'Accise 
National Parole Board 


Revenue Canada, Taxation 


Automatic Day Parole Review Provisions Access to Information and Privacy /// 


Excise Revenue Collections /// Perception 


of Bill C-67 Gointly with Correctional 
Service Canada) /// Dispositions du 
projet de loi C-67 relatives à l’examen 
automatique en vue de la semi-liberté 
(conjointement avec Service 
correctionnel Canada) 


National Research Council Canada 


Biological Sciences 
Mechanical Engineering 
National Aeronautical Establishment 


Natural Sciences and Engineering 
Research Council of Canada 


Canadian Microelectronics Corporation 


Public Service Commission of Canada 


Appeals 


Employment Equity /// Rapport sur 
l'évaluation des services du programme 
d'équité en matière d'emploi dans la CFP 


Investigations Function 


_ Recruitment and Referral 


Public Works Canada 


Architectural and Engineering Services 
Technology Development /// Evaluation 
des possibilités technologiques des 
services d’architecture et de génie 


The Performance of General Contractors 
/// Le rendement des entrepreneurs 


Regional Industrial Expansion 


Microelectronic Centres 
Native Economic Development Program 
Small Business Loans Act 


Technology Opportunities in Europe 
Program 


Accès à l'information et protection des 
renseignements personnels 


Audit T1 and T2: Individual and 
Corporate Returns /// Vérification des 
déclarations d’impét des travailleurs 
indépendants et des corporations 


Independent Survey of the Public 
Enquiries Program 


Management Services Branch Review 


Taxes in Dispute Legislation /// Les 
mesures législatives concernant les 
impôts contestés 


University Recruitment of Auditors /// 
Campagne de recrutement universitaire 
d’auditeurs 


Royal Canadian Mounted Police 


Marine Services 


Technical Security 


Secretary of State of Canada 


Citizen Registration and Promotion /// 
Une évaluation de l’enregistrement de la 
citoyenneté et de la promotion du 
civisme. Volume 1: Sommaire - 
Principaux résultats et conclusions 


Evaluation of the Official Languages 
Services /// Evaluation des services en 
langues officielles 


Evaluation of the Police-Minority 
Initiative in the Multiculturalism Sector 


Human Rights 


Native Friendship Centres /// Evaluation 
du programme des Centres d’amitié 
autochtones (Résumé) 


Native Social and Cultural Development 


Transport Canada 


Veterans Affairs Canada | 
| 


Social Sciences and Humanities 
Research Council of Canada 


Aid to Small Universities 
Family and Socialization of Children 


Tourism Pilot Project 


Revenue Canada, Customs and Excise 


Adjudications /// L'arbitrage 
Human Context of Science and 


Assessment Programs and Valuation /// Technology 


Etude d’évaluation des programmes de 


cotisation Strategic Programs 


Solicitor General Canada 


Secretariat Interns Program 


Centres of Criminology 


Women in Conflict with the Law 


Statistics Canada 


Agriculture Statistics 
Canadian Centre for Justice Statistics 
Consumer Price Indices 


Corporations and Labour Unions Re 
Act (CALURA) 

Demography 

Input/Output Statistics Program 
Monthly Industry Output Measures /// 


Programme de mesure et d’analyse des 
industries 


Small Area and Administrative Data | 
Program /// Programme des données 
régionales et administratives 


Supply and Services Canada 


Consumer Products Procurement 
Corporate Data Management 

EDP Buy-For-Lease Program 

Manitoba Pilot Project | 
| 
Office of Primary Interest: Governmen | 
Information Collection 1 


Ministerial Correspondence System 


Small Business Policy | 
Software Exchange Pilot Project | 


Mid-Term Review (1987): 
Canada/Manitoba Transportation | 
Subsidiary Agreements | 


Public Harbours and Ports / valuation 
des havres et ports publics 


Veterans Independence Program 


War Veterans Allowance/Civilian War 
Allowance 


M 
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ES Annual 


>onference 1990 


‘he Canadian Evaluation Society (CES) held its 
ith annual conference from May 14 to 16, 1990, in 
foronto, with over 500 participants attending from 
cross the country. The theme of Conference 90 
vas “New Perspectives on Evaluation for the 90s. 


the first day was devoted entirely to workshops, 
our of which were of an all day-format, the rest 
unning either for the morning or the afternoon. 
fopics for the former included cost-benefit analysis, 
lesigning health promotion surveys, an overview of 
nethods and models, and methods for health care 
issessment. The latter sessions also exhibited a 
road range of topics including focus groups, 
lanning useful evaluations, multi-attribute 
inalysis, computer-assisted telephone surveys, 
etting the best from consultants and advanced 
yuestionnaire design. 


The first plenary session featured Dr. Ross F. 
Sonner, Immediate Past President of the 
American Evaluation Society and Associate 
-rofessor and Research Psychologist, University of 
Jalifornia at Irvine. Dr. Conner presented two very 
lifferent, interesting and illustrative case studies. 
The first case was an evaluation of the Citizen 
Dispute Settlement Program in Florida, the other 
was of an AIDS Prevention Program for Latino 
nigrant workers. From these studies, Dr. Connor 
was able to draw out a number of opportunities for 
valuation in the future. He encourages taking a 
:lobal perspective, incorporating technological 
change in increasing the spread of ideas, benefiting 
rom diversity and the possibility of reinventing or 
re-energizing evaluation. Dr. Connor foresees many 
future opportunities to apply new methods such as 
focus groups, to work with new types of programs 
for example his case studies) and to work more 
closely with clients and program developers, 
resulting in more useful and valued evaluations. 
He encourages more contact with program clients 
to better understand their perspective, the 
program and its processes, keeping in mind the 
cultural differences of this group. 


Program Evaluation 


Wally Beevor, the Assistant Deputy Minister of 
Learning Program Division, Ontario Ministry of 
Education, spoke of the challenges of making 
learning more meaningful and active and of 
encouraging students to adapt to new cultural and 
technological literacies. He also spoke of the different 
philosophies of school and student performance in 
the U.S. and Canada and the challenge of evaluating 
achievement in a meaningful way. He regards 
accountability as the credible communication of what 
one is doing that is of significance and value, and 
feels the greatest challenge is to steer a reasonable 
course between complexity and accountability. 


The concurrent sessions covered a broad range of 
interesting topics such as quality assurance, 
evaluation of regulatory programs, links between 
policy and evaluation and evaluation as a means 
for managing in a health care and services 
environment. 


Outgoing President Ronald C. Corbeil, Office of 
the Comptroller General, presided over the Annual 
General Meeting and presented the Membership 
Committee’s and Administration Committee’s 
reports. Current membership stands at 
approximately 1200. Committee reports were also 
presented for Professional Development, 
Communications and Regional Services. Chapters 
have or are being established in all provinces or 
territories. Regional reports indicate the majority 
of chapters held a number of activities such as 
workshops, luncheons, presentations and members’ 
surveys during the last year. Other business 
conducted included the presentation of financial 
statements, reappointment of the auditors and the 
election of the new Council. John McDermott, 
Director of Program Development and Quality 
Assurance for Alberta’s Social Services Ministry is 
the new President. 


The final day of the conference commenced 
with a plenary session in which Dr. Michael 
Rachlis, M.D. used a case study of an evaluation 
on the use of cholesterol screening in improving the 
effectiveness and efficiency of health care. Through 
evaluating the cost-effectiveness of the screening 
and other programs, recommendations emerged to 
target the screening program to high-risk persons 
and to promote changes in food processing, diet 
and lifestyles. Dr. Rachlis pointed to the 
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desirability of dispensing with expensive, high-tech 
interventions and relying more on individual 
doctors’ efforts. This session concluded with 
workshops on such topics as the role of evaluation 
in reforming administrative policies, evaluation of 
training programs, evaluation in community 
mental health and in the federal government. 


At the President’s luncheon, John McDermott, 
incoming President and a long-time active 
supporter of the CES, addressed the audience. 
John first thanked the retiring Council members 
and welcomed the new ones on board. His theme, 
“CES: Facing the Challenges of the 90s”, 
encompassed the society’s activities and goals. He 
sees its goals as increasing membership, giving 
better staff support to the chapters, raising the 
society’s profile and providing more professional 
development activities. He looks to the 90s to bring 
increased relevance and credibility to evaluations. 


Four awards were presented at the luncheon. 
Kylie Hutchinson was presented with an award 
cheque for the best Student Evaluation Essay (a 
proposal to evaluate a pamphlet on heart disease). 
Ilsa Blidner received the first award for 
Exemplary Service to the CES and was praised by 
the President for her fine work. The President then 
presented Dr. Greg Mason, of Prairie Research 
Associates and the University of Manitoba’s 
Economics department, with the CES Award for 
Contribution to Evaluation in Canada. Dr. Mason, 
a past President, is actively involved in chapter 
work and the promotion of program evaluation. 
Finally, the CES Public Sector Award was 
presented (in absentia) to Kenneth Dye, the 
Auditor General of Canada, for his contribution to 
the establishment. of the program evaluation 
function within the federal government. 
Maria Barrados, a Principal with the Office of 
the Auditor General of Canada, accepted the award 
on his behalf, though we were able to witness 
Mr. Dye’s videotaped acceptance of the award. 


Early afternoon workshops followed, then the 
conference closed with a final plenary session 
entitled “New Perspectives on Evaluation for the 
90s”. Guy Leclerc, Deputy Comptroller General, 
stressed the changing nature of the environment in 
which government evaluation is undertaken and 
said that it will play a key role in reporting on 
effectiveness. Evaluation will have to become a tool 
for all stakeholders, will have to deal with 
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implementation issues, will have to be bol 
practical and rigorous and will have to develop 
diversity of approaches without compromising 0 
basic principles of evaluation. Maria Barradi 
also stressed diversity, flexibility and t) 
importance of evaluation in measurij 
effectiveness. Over the years the Auditor Genel 
has made observations on departmental evaluati) 
functions and the timeliness, quality and clarity’ 
presentation of studies. Ms. Barrados loo| 
forward to increasingly effective and usef 
evaluations, given the foreseen increase 
accountability demands. David Black, a manag 
with ARA Consulting Group Inc., sees globalizatit 
increasing the competition for resources in | 
areas, which may force groups and individuals i 
larger strategic alliances, with much more fluid) 
dynamic programs emerging. He sees the notion 
the traditional bottom-line changing, with 
consequential need to find new ways of looking 
it, accompanied by a new sense of speed ai 
timeliness. Greg Mason spoke of the challenges 
assisting clients in selecting from a wide variety] 
information, in becoming more cross-cultural 
our evaluations and of better marketing o 
services to the private sector, which is increasing 
concerned with efficiency and effectivenes 
Discussion ensued on the standards for feder 
government evaluations and audits. The need 
make the information produced more valuable 
decision-makers was also highlighted. 
| 


As in previous years, federal evaluators a1 
officials made a strong contribution to tl 
conference. Contributing to various workshof 
panels and other sessions were: Nan¢ 
Higginson, Ian Kelley, Carol Motuz (AGI 
Kenneth Howard Tiedemann (CCAC); Dot 
Borrowman, Iris Bradley (CSC); Shelle 
Borys, Bill Graham (COM); Karen Traver: 
(DFO); Serge Bertrand, Nand Tandan (EI 
Alain Barbarie (EMR); Beverley Clarksc 
(INAC); Dorothy Hepworth (JUS); Chan A 
(LAB); Sheila Faure (NPB); Maria Barrad 
(OAG); Ronald C. Corbeil, Guy Lecler 
Cameron McQueen and Toru Uno (OCG); Ma: 
Gusella (PCO); Nicole Mendenhall, Ron McG 
(RC-CE); Barry Leighton, Stan Divorski (SG 
Adele Copti-Fahmy, Jennifer Stewar 
Jerome Thauvette (SSC); Chuck Lyon (S( 
Mirelle D’Auray-Spattz, Heather Hopwoo 
Lee McCormack, Wayne McCutcheon, Ma: 
McLaren, Stephen Tsang and Alan Winbe 
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TBS); Joanne Coughlin, Jon Fisher and 
Michael Lotan (TC); Paula Thompson 
VAC).The 12th Annual Conference of the CES will 
ye held in Vancouver, B.C., from May 5-8, 1991, at 
he Westin Bayshore. Its theme will be “The 
olitics of Evaluation”. & 


Allison Fader 


\llison Fader is an Evaluation Analyst with the 
Program Evaluation Branch, OCG. 


Hearings on Program Evaluation 
y the Senate Committee on 
National Finance 


\ Parliamentary spotlight has been focused on 
valuation at recent hearings of the Senate 
Standing Committee on National Finance in 
sonnection with its order of reference to examine 
the 1990-91 Estimates. The Senators’ interest in 
valuation was prompted by a controversial draft 
baper by Prof. Douglas G. Hartle of the 
University of Toronto entitled Increasing 
Government Accountability: A Proposal that the 
Senate Assume Responsibility for Program 
Evaluation. Professor Hartle is a recognized 
authority on the expenditure budget process and 
was a senior official with the former Planning 
Branch, Treasury Board Secretariat. 


From the end of March through mid-June, the 
Committee held lively sessions most weeks on the 
strengths and weaknesses of departmental 
evaluation with: Douglas Hartle; Kenneth Dye 
and Maria Barrados from the Office of the 
Auditor General (OAG); Eleanor Chelimsky, 
Assistant Comptroller General of the United States 
(Program Evaluation and Methodology Division), 
United States General Accounting Office; senior 
managers and evaluators from the Departments of 
the Secretary of State and Fisheries and Oceans; 
Kenneth Watson, an evaluation consultant with 
Rideau Research Associates Ltd.; John 
McDermott, the current President of the 
Canadian Evaluation Society and Director, 
Program Design and Quality Assurance, Alberta 
Family and Social Services; and J. Andrew 
Macdonald and Guy Leclerc from the OCG. 


Program Evaluation 


A related meeting of note was the well-attended 
session on May 3, sponsored by Canadian Evaluation 
Society (National Capital Chapter) and the Library 
of Parliament on Program Evaluations: a Useful 
Information Source for Parliamentarians? Speakers 
were: Senator Lorna Marsden, a member of the 
Committee on National Finance and Chair of the 
Committee on National Finance and Chair of the 
Committee on Social Affairs, Science, and 
Technology; Eric Adams of the Research Branch, 
Library of Parliament; Maria Barrados of the OAG; 
and John Mayne of the OCG. 


The Committee proceedings are available in the 
Hansard reports. Following more hearings on 
evaluation in the fall, the Committee will report to 
the Senate on program evaluation in the federal 
government and on possible improvements. Stay 
tuned. Æ 


Cameron McQueen 


Cameron McQueen is Senior Policy Analyst with 
Program Evaluation Branch, OCG. 


ee See Polytoshnié nstitute, di | 
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Program Evaluation 
Orientation 
Workshop (English) 


September 23-28, 1990 
Donald Gordon Centre 
Kingston, Ontario 


This five-day in-residence workshop de an 
overview of the phases of the evaluation process. 
By means of case studies, participants will have 
the opportunity to complete the steps in the 


evaluation process and receive feedback on their 
efforts from their colleagues and session leaders. 
Newly appointed evaluation officers are 
encouraged to participate in this core workshop. 


It covers material considered to be preparatory 
to other PEB seminars and eee il be ae 


Content 


The context for program evaluation - 
The program component profile 


The evaluation assessment 
— generating a list of questions 


— choosing appropriate evaluation 
approaches 


developing evaluation options 


writing the assessment report 
Evaluation frameworks 


The evaluation study : 
— considerations in data collection 


— producing findings and drawing 
conclusions 


— the format and content of the executive 
summary and study report 


Implementation of results 


To Register 
Please contact Joan Cater at (613) 957-7180. 


‘ 
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OCG Practicum Sessions 
Discuss How to Achieve 
Cost-Effectiveness in the 
Federal Public Service 


For the ninth consecutive year, the OCG organiz 
four Practicum sessions. The target cliente 
consisted of managers in finance, internal aud 
and program evaluation. The half-day sessio! 
were held on October 23 and December 1, 198 
and February 5 and April 4, 1990, at ti 
Government Conference Centre. The organizatil 
of this ninth season was the responsibility of ti 
Program Evaluation Branch (PEB). | 


The 1989-1990 Practicum season was devoted | 
studying established ways and means of maki 
programs more cost-effective. The practical aspec 
of program cost-effectiveness were explored in t] 
following sessions: cost-effective administrati 
management and control; demonstrating cos 
effective programs; improved performance throu; 
better financial information; and explorit 
program delivery options. 


A large number of guest speakers from tt 
departments and the central and other agenci 
helped to make Practicum a success. We would lil 
to thank the following people for the 
participation: Robert Letellier, Hele 
Mcdonald (Com); George Neufeld (Coopers 

Lybrand); Serge Bertrand, Ian Midgley (EIC 
Marc Foster (EAITC); Mare Van Den Bort 
(F&O); Roger Beauparlant (GTA); John. 
McCrea (ISTC); Graeme M. Kirby (NCC); Je 
Sarazin (ND); Bernie A. Gorman, Guy Lecler 
John Mayne, Barry Rosenfeld, Laura M 
Talbot-Allan, George Willis (OCG); Louis 
Saint-Laurent (PW); Nicole Mendenhal 
R. Struthers, E.J. Warren (RC-CE); Dougls 
Newman (RC-T); Dennis Kam, Richard Pato! 
Michael Stephens, Sam Wex (TBS). @ 


Ibolya Galambo 


Mme Galambos est agente de recherce et d’analys 
a la Direction de l’évaluation de programme 
du BCG. 
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Affaires extérieures et 
Commerce extérieur Canada 

Bill van Staalduinen 

Directeur, Division de l'évaluation 
et de la revue des ressources 


Affaires indiennes et du 
Nord canadien 

Marie-France D’Auray-Boult 

Directrice intérimaire, 
Direction de l'évaluation 


Agence canadienne de 
développement international 

Jean Quesnel 

Directeur, Division de l'évaluation 
des programmes 


Agence de promotion économique 
du Canada atlantique 

David Tripp 

Directeur général, Évaluation FAX 
et Revue 


Agriculture Canada 

Carol Motuz 

Directrice intérimaire 

Division de l'évaluation des 
programmes 


Anciens combattants Canada 

Dick St. John 

Directeur, Évaluation des FAX 
programmes 


Approvisionnements et 
Services Canada 

Jennifer Stewart 

Directrice, Direction de l'évaluation 


Archives publiques du Canada/ 
Bibliothèque nationale du Canada 
Françoise Houle 
Directrice générale, 
Direction des politiques 


Banque fédérale de 
développement 

Jacques Lagacé 

Vice-président 

Vérification interne et évaluation 
des programmes 


Bureau du surintendant des 
institutions financières 
Neville Grant 
Directeur exécutif, Secteur 
des services de gestion 


Bureau du vérificateur général 

Ted F. McNamara « 

Vérificateur général adjoint 

Évaluation des programmes et 
vérification interne 


Centre canadien d'hygiène et 
de sécurité au travail 

Kash Manchuk 

Directeur, Finances, publications FAX 
et administration 


Centre canadien de gestion 
Louise Durocher 
Gestionnaire en évaluation 


Centre de recherches pour 
le développement international 
Douglas Daniels 
Directeur, Bureau de planification 
et d'évaluation 


Commissariat aux langues 
officielles 

Yves Breton 

Conseiller 


Commission canadienne du lait 
Charles C. Birchard 
Analyste supérieur en politique 


Commission de contrôle de 
l'énergie atomique 

Jim Elks 

Directeur intérimaire, 
Planification et coordination 


Commission de la Capitale 


Vérification de gestion et 
évaluation des programmes 


Commission de la fonction 
publique du Canada 

Suzanne Peters 

Directrice, 

Évaluation de programmes 

Direction de la revue 


Commission canadienne des 
droits de la personne 
Debra Young 
Directrice 
Directioh de la politique et 
de la recherche 


Commission des relations de 
travail dans la fonction publique 

Louis Labrecque 

Directeur, Administration 


Commission nationale des 
libérations conditionnelles 

Sheila Faure 

Gestionnaire, Evaluation et 
statistiques 


993-5829 
FAX 990-9021 


994-4311 
FAX 994-1283 


997-0148 
FAX 953-6356 


1-506-857-6515 
1-506-857-7403 


994-1991 
FAX 994-1782 


1-902-566-8017 
1-902-566-8051 


956-0892 
FAX 956-4962 


992-0660 
FAX 992-9010 


1-514-283-7300 


990-1718 
FAX 952-8219 


925-4601 


1-416-572-4401 
1-416-572-2206 


992-8061 
FAX 995-0331 


598-0593 
FAX 238-7230 


995-0815 


998-9490 
FAX 998-4492 


995-6535 
FAX 995-5086 


239-5082 
FAX 239-5098 


992-8592 
FAX 995-1593 


990-1825 


994-7474 
FAX 995-4380 


Agents de liaison ministériels pour 
l'évaluation des programmes 


. 


Conseil de la radiodiffusion et 
des télécommunications 
canadiennes 

Denis O'Rourke 

Directeur, Vérification intégrée 
et évaluation 


Conseil de recherches en 
sciences humaines 

Robert Hanson 

Coordinateur, Direction de 
l'évaluation des programmes 


Conseil de recherches en 
sciences naturelles et en génie 

Jocelyn Lillycrop 

Gestionnaire, Evaluation des 
programmes 


Conseil de recherches médicales 
Michael O'Brecht 
Evaluation des programmes 


Conseil des arts du Canada 
Peter Brown 
Directeur adjoint principal et trésorier 


Conseil du Trésor du Canada 
Cabinet du secrétaire du Conseil 
du Trésor 
Doug Redden 
Analyste principal, 
Politique intégrée 


Direction de la politique 
du personnel 

André Robert 

Directeur, Groupe de l'évaluation 
et de la revue 


Direction de la politique 
administrative 

Alan Winberg 

Directeur, Division de la 
planification et de l'évaluation 


Direction des langues 
officielles 

Marcus Hotz 

Directeur, Evaluation des 
programmes 


Conseil du Trésor du Canada, 
Contréleur général 

Evelyn Levine 

Directrice, Gestion intégrée 


Conseil national de recherches 


Chef, Evaluation des programmes 


Consommation et Corporations 
Canada 

Bruno Bonneville 

Directeur adjoint intérimaire, 
Evaluation des programmes 


Corporation commerciale 
canadienne 

F. Owen Kelly 

Contréleur 


Cour supréme du Canada 
Irene M. O'Connor 
Chef, Services financiers 


Défense nationale 

Paula Skippon 

Directrice générale, Évaluation 
des programmes 


Défense nationale (CST) 

Donald Drew 

Directeur 

Bureau des services d'évaluation 
et de la revue 


Diversification de l’économie 
de l'Ouest 

Guy Brunet 

Directeur général, 

Evaluation et service à la 
haute gestion 


Emploi et Immigration Canada 
Tan Midgley 
Directeur général, Evaluation 

des programmes 


Energie, Mines et Ressources 
Canada 

Alain Barbarie 

Directeur, Direction de l'évaluation 
des programmes 


Environnement Canada 

Pat Gore 

Gestionnaire, Evaluation des 
programmes . 


Directeur, Vérification et 
évaluation 


Gendarmerie royale du Canada 
Donald Coates 
Surintendent principal, Service 

à la coporation 


Industrie, Sciences et Technologie 
Canada 

Byd McBain 

Directeur, Direction de l'évaluation 


Investissement Canada 

C. Edward Hitsman 

Directeur, Direction des services 
intégrés 


997-9414 
FAX 994-0218 


992-5148 
FAX 992-1787 


995-5911 
FAX 992-5337 


954-1817 
FAX 954-1880 


598-4311 


957-2419 
FAX 996-2690 


952-3141 
FAX 996-2690 


957-0743 
FAX 996-2690 


952-2860 
FAX 996-2690 


957-7071 
FAX 952-8536 


993-4758 
FAX 954-5386 


953-6735 
FAX 997-2721 


996-0116 


996-7662 


996-4251 
FAX 992-0528 


991-7315 
FAX 991-7500 


952-3933 
FAX 952-9384 


994-2406 
FAX 997-7329 


996-2464 
FAX 996-6424 


997-6946 
FAX 953-2107 


997-1107 
FAX 995-1037 


993-3633 
FAX 993-0216 


954-9633 
FAX 954-1894 


995-3452 
FAX 996-2515 


Ministére des Communications 

Francine Chabot-Plante 

Directrice, Direction de l'évaluation 
des programmes 


Ministère des Finances 

David Sewell 

Directeur 

Division de l'évaluation et 
des mesures fiscales 


Ministère de la Justice 

Michel Vallée 

Directeur intérimaire, 
Evaluation des programmes 


Musées nationaux du Canada 

Wally Kozar 

Secrétaire général adjoint, 
Service à la corporation 


Office national de l'énergie 
Claire Scott 
Directrice, Planification et revue 


Office national du film 
Claude Parent 


990-4735 
FAX 952-1208 


992-4309 
FAX 992-4450 


993-5003 
FAX 992-6340 


954-4250 


990-6845 
FAX 990-7900 


1-514-283-9251 


Coordonnateur à la FAX 1-514-496-1646 


planification et à l'évaluation 
de programmes 


Office national des transports 
Mike Andrecheck 
Directeur 
Direction générale des 
services financiers 


Pêches et Océans 

Robert Bergeron 

Directeur, Direction de l'évaluation 
des programmes 


Programme national recherche 
et sauvetage 

David Walters 

Directeur, Vérification et évaluation 


Protection civile Canada 
John Pritchard 
Directeur, Évaluation et analyse 


Revenu Canada - Douanes 
et Accise 

Nicole Mendenhall 

Directrice, Direction de l'évaluation 
des programmes 


Revenu Canada - Impôt 

André Vallières 

Directeur intérimaire, 
Direction de l'évaluation 
des programmes 


9 ~~ 
Santé et Bien-étre Canada 
Bud Long , 
Directeur, Evaluation des 
programmes 


Secrétariat d'État 
Kerry Johnston 
Directeur, 
Évaluation des programmes 


Service canadien du 
renseignement de sécurité 

Joanne Stoody 

Chef, Examen des programmes 


Service correctionnel Canada 
Doug Borrowman 

Directeur intérimaire, 
Évaluation et recherche 


Société canadienne d’hypothéques 
et de logement 

Philip Brown 

Directeur, Division de l'évaluation 
des programmes 


Société du crédit agricole 
Michel Messier 
Chef, Évaluation des programmes 


Gestionnaire, 
Évaluation des programmes 


Société Radio-Canada 

Ron Sweeney 

Directeur, Coordination dela 
planification et du l'évaluation 


Solliciteur général du Canada 
Jean Filion 
Sous-solliciteur général adjoint 
Direction de la planification et 

de la gestion 


Statistique Canada 

Martin Hubbard 

Directeur, Division de l'évaluation 
des programmes 


Téléfilm Canada 
Robert Armstrong 
Recherces et Analyses 


Transports Canada 

Norm Steinberg 

Directeur, Évaluation des 
programmes 


Travail Canada 

Denise Roberge 

Directrice, Services ministériels 

Travaux publics Canada 

Barry DRE 

Directeur, Évaluation des 
programmes 


997-1260 
FAX 953-8623 


993-1937 
FAX 996-9055 


996-1696 
FAX 996-4452 


991-7071 
FAX 996-0995 


954-7881 
FAX 952-6770 


954-6042 
FAX 957-7476 


957-3889 
FAX 957-8910 


994-1224 
FAX 997-7836 


957-8244 
FAX 952-1476 


995-9000 
FAX 995-3603 


748-4659 


598-2870 


738-6592 


991-2942 
FAX 990-9077 


951-8291 
FAX 952-9099 


1-514-283-6363 


993-4418 
FAX 954-1993 


994-0728 
FAX 997-1664 


736-2080 
FAX 998-9603 
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Report on the Program Evaluation Directors’ 
Conference at Montebello 


June 12-13, 1990 


On June 12 and 13, directors of eval- 
uation from major federal depart- 
ments and officials from the Office of 
the Comptroller General (OCG) met 
in Montebello to discuss a number of 


. topics of current concern to the fed- 


eral evaluation community. This was 
the first time directors had had the 
opportunity to spend that length of 
time together in a less formal setting. 
Some 35 heads of evaluation, several 
substitutes and 9 officers from the 
OCG participated. 


Prior to the event, suggestions for top- 
ics were solicited from directors and 
they were asked their preference for 
participation in discussion in small 
groups. The format for the event after 
an opening talk by Mr. Andrew 
Macdonald, Comptroller General of 
Canada, in the afternoon of the 12th, 
was to break into small groups to 


discuss the topics and then to present 
the conclusions of the groups in 


plenary. 


Mr. Macdonald welcomed the partici- 
pants and talked briefly linking 
PS2000 and program evaluation. He 
reviewed the aims of PS2000 and 
discussed some of the recommenda- 
tions that were being proposed with 
respect to budgetary simplification, 
streamlining of financial manage- 
ment and revitalization of human 
resource management. He empha- 
sized the need to reinvigorate the 
public service and reinforce its role as 
a provider of service to the public. In 
large part this can occur by empower- 
ing public servants with the tools they 
need to get the job done but also hold- 
ing them to account. 
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In that light, Mr. Macdonald challenged the community 
to seize the opportunity to play a key role in contribut- 
ing to the results-focused management envisaged by 
PS2000. Evaluation can provide the missing “bottom 
line” of the private sector, helping managers to acquire 
better information about the performance of their pro- 
grams, providing a corporate challenge to the status quo 
and a means to demonstrate the performance of pro- 
grams. He particularly noted the unique role of program 
evaluation in assessing the effectiveness of services to 
the public and how these could be measured. 


“I expect the Program Evaluation Branch to 
lead the evaluation community in responding to 
these opportunities. It should, in close coopera- 
tion with the community, identify the recom- 
mendations of PS2000 which can involve pro- 
gram evaluators, formulate appropriate 
strategies and implement them across govern- 
ment. We must help meet the challenges of 
PS2000 faced by departmental management 
and central agencies. The increased focus on 
management for results and accountability 
affords program evaluation a unique opportu- 
nity to assist management in responding to the 
new management culture in the forthcoming 
decades.” 


Eight topics were discussed in small groups and in ple- 
nary by participants and a summary of each of the dis- 
cussions is given elsewhere in this Newsletter. As a 
result of the discussions, the OCG will be undertaking 
or reinforcing a number of activities. 


Several of the subjects discussed will be addressed in 
GPEP ’90 this fall: Barriers to Productive and Effective 
Evaluation, Demonstrating Program Cost-Effectiveness 
and Evaluation in Program Performance Measurement 
and Reporting. Concerning Demonstrating Evaluation 
Utilization we will be clearly describing the expecta- 
tions we have regarding utilization and are planning a 
one day event on utilization later this year. We are also 
encouraging departments to take the time to document 
and communicate, both internally and externally, good 
examples of evaluation utilization. 


We are discussing the evaluation and audit issue with 
the Audit and Review Branch here at the OCG and 
developing some material to better communicate the 
roles and comparative advantages of each. In the 
Human Resource area, we will be issuing in the fall a 


detailed profile of the federal evaluation community 
working with the evaluation community to develop, 
longer term strategy to deal with the human resouri 
issues of the 90s. The new evaluation policy now bei 
finalized will highlight the role of line managers in eval 
ation while re-emphasizing the need for objectivity. It 
also discuss the usefulness of a link between evaluati 
and policy development. = 


Guy Lecle 


Guy Leclerc is Deputy Comptroller General of Canad 
Program Evaluation Branch, OCG. 


The following are the topics discussed at the 
Directors’ meeting of June 12 and 13, 1990: 


L Human Resource Management (Community 
Development) 


: Demonstrating Evaluation Utilization 


. Evaluation in Program Performance 
Measurement and Reporting (IMAA) 


. Demonstrating Program Cost-Effectiveness — 


. Objectivity and the Role of Line Management in 
Evaluation 4 


. Program Evaluation and Internal Audit - 
. Barriers to Productive and Effective Evaluer io n ; 


. Evaluation and Policy Development 
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|. Human Resource 
Management (Community 
Development) 


ssues 


‘he group reviewed the current status of evaluation staff 
nd identified some present and future skill require- 
1ents as well as critical factors for our future as a com- 
aunity. 


Discussion 


he group began by reflecting on the quality of current 
taff. Generally speaking, it was seen as being “good to 
xcellent” in terms of both potential replacements (for 
rectors) and newer staff, although a shortage of experi- 
nced evaluators was noted. 


his discussion led to consideration of one of the main 
sues: availability of appropriate skills and abilities. The 
roup began by deliberating on whether or not evaluation 
a good career move, and at what stage in one’s career. 
here is no common perception across government that it 
a “high flyer activity”. However, there was a feeling 
lat evaluating is the “best management training avail- 
ble” and that it is also a way to contribute to or influ- 
nce departmental events. The level of support for the 
inction in a department is a factor: work is sought out in 
nits seen as being influential. Accordingly, good solid 
eople from program areas are now approaching some 
valuation units seeking employment. There was mixed 
pinion as to the usefulness of formal assignments pro- 
rams; while the people were frequently well qualified, 
1ey often did not stay long enough to return the invest- 
lent made in them. Universities are seen as an obvious 
urce of good people (some are beginning to offer special- 
es in evaluation) but the need for training will continue. 


he impact of the use of consultants, especially in terms 
the mix of skills required, received some consideration. 
, was noted that even when consultants are frequently 
sed, good internal evaluators, who are aware of priori- 
es, the culture, etc. are needed and will continue to be 
eeded. 


he plenary session was presented with four critical fac- 
rs related to our future as a community: the image of 
valuation, a steady supply of good candidates, profes- 


Program Evaluation 


sional training and development, and the role of the 
OCG. With respect to human resource management in 
the longer term, responsibilities were assigned, first and 
foremost, to departmental management. However, other 
parties are also responsible. As the career of the average 
evaluator is short, responsibility for the longer-term 
interests of the community rests with the OCG. 
Consulting firms were also identified as possible contrib- 
utors, particularly in the training area (it was noted that 
they continue to benefit from the evaluation function). 
The Canadian Evaluation Society also continues to play a 
role in the area of professional development. m 


2. Demonstrating Evaluation 
Utilitzation 


Issues 


This group considered the importance of demonstrating 
the utilization of evaluation, what has been done so far 
and what more could be done. 


Discussion 


It was noted that some academics and some bureaucrats 
are sceptical or unaware of the benefits of evaluation. 
This can be traced to a lack of awareness of the many 
uses already made of evaluation or to a misunderstand- 
ing of what benefits evaluation is capable of delivering. 


The endurance of the function (especially in comparison 
to other “alphabet soup” initiatives) and the confidence of 
many directors that they are valued parts of manage- 
ment counter somewhat the outside scepticism. 
Nevertheless, there was still some feeling of sensitivity 
on this point. Should we be marketing only to our most 
important clients or should we strive for outside under- 
standing as well? 


Evaluators are especially conscious of methodological 
problems in demonstrating the usefulness of evaluation. 
As one example, we understand our own possible bias on 
this question. As another, attributing benefits to evalua- 
tions is often extremely difficult or sensitive. 
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Suggestions 


Some suggestions were made: 

¢ to make better use of Part IIIs; 

¢ to develop a professional marketing strategy; 
e to use annual reports; 


° to take more responsibility for outside (press) cover- 
age of evaluations; 


° to document more case studies; and 


¢ to elicit endorsements from important users. 


Work already done by the OCG in documenting utiliza- 
tion was recognized as good background although it obvi- 
ously had not settled the issue, nor perhaps had been 
communicated to best advantage. 


The plenary discussion underlined that although plenty 
of use had been made of evaluation, and much of it docu- 
mented, and some of it communicated, there was still a 
gap between the perception and the expectation of some 
observers. This could be countered partly through better 
demonstration of usefulness but the other part of the 
problem was erroneous or unrealistic expectations. After 
all, what fraction of evaluations should lead to highly 
visible reallocations of resources? 


In general, it was felt that the process benefits and the 
management assurance or accountability benefits of eval- 
uation have been ignored by critics and underplayed by 
advocates. = 


3. Evaluation in Program 
Performance Measurement 
and Reporting (IMAA) 


Issues 


Two separate syndicates discussed matters such as the 
current impression of IMAA in departments; what it has 
contributed to in terms of opportunities for managerial 
innovation, clarity about future directions and expecta- 
tion for performance; and how evaluation can best work 
with other departmental staff in support of IMAA 
objectives. 


groups noted that current impressions in departme 


Discussion 


ences in terms of the overall acceptance and impac 
and expectations for, the IMAA initiative. In one der! 


ously existing reports were used and (apparently) fo 
acceptable. Internally, new authorities were not foun 
have been delegated below the director general level 


ment, the evaluation unit played a large role in e 
deliberations leading up to the signing of the MOU. 


There was a general feeling that the IMAA initiative : 
contributed to greater expectations about program 
formance, particularly due to the work on OPFs. © 
again though, the degree of involvement of the evalua’ 
units varied. Perhaps naturally so, the perceptior 
IMAA by evaluation personnel may be influenced by 
extent to which the evaluation unit is involved in 
department’s efforts. In some departments, work on 
IMAA initiative was seen to have raised the profile of 
evaluation unit. In a joint presentation in plenary, 


revolved around a general lack of benefits realized fi 
IMAA and a wait-and-see approach, particularly si 
the announcement of PS2000. This lack of momen 
had also been noted on the part of Treasury Board. 
one participant stated, his role with respect to IMAA : 
to “advise the DM when he is in danger of being outs 
the Corporate pack”. 


The OCG was advised that the more support it provi 
for the initiative, the better the initiative would be 
was also asked to “explain the value of evaluation 
Program Branch (TBS)”. = | 


4. Demonstrating Program 
Cost-Effectiveness 


Issue 


The focus in this area is on alternatives to existing p 
gram arrangements, specifically their actual or potent 
comparative performance in relation to present p 
grams. More precisely, the issue is how evaluation ( 
better help managers to: (i) assess the continued rati 
ale of programs (sometimes with a comparison with 1 
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10-program alternative), (ii) identify alternative program 
jesigns or delivery mechanisms and (iii) compare and 
jocument the feasibility and cost-effectiveness of alterna- 
‘ives. 


[he technical terms, “efficiency”, “cost-effectiveness anal- 
sis”, “cost-accounting”, and “benefit-cost analysis”, do 
rot convey unambiguously to managers or evaluators a 
specific type of decision framework, accountability con- 
ext or a particular analytic approach. Evaluators must 
earn to communicate more clearly in this area without 
elying on unexplained jargon. 


Jiscussion 


articipants felt that much “cost-effectiveness” or “alter- 
latives” evaluation work had been done in federal evalu- 
ition, more on delivery alternatives and less on program 
lesign alternatives. Much of this work has been fed into 
nternal departmental program rearrangements and 
esource reallocations, with less use in TB or other 
Jabinet committee deliberations. Participants wanted to 
ee an increase in the use of such evaluation work by 
Jabinet committees. There is little interest by managers 
Nn purely ex post accountability analysis of alternatives 
vhich is not intended to lead to program adjustments. In 
Many cases, evaluation provides the occasion and the 
nalytic discipline for departmental consideration of 
lternatives of which managers are well aware. Cost- 
ffectiveness analysis of alternatives often bridges the 
* post and the ex ante perspectives and provides federal 
valuators opportunities to do formative work. 


he depth of alternatives analysis provided by evaluation 
aries a great deal. In some departments, other line or 
taff groups take the lead here with assistance from the 
valuation group. In others, evaluators’ knowledge and 
n-depth documentation of the cost-effectiveness of alter- 
atives may involve them in policy development, which 
[as seen as something to be encouraged. 


imilarly, departmental evaluation practice concerning 
ationale issues varies a great deal, depending on such 
actors as the politics of individual programs, deputy 
ead orientation (policy adviser or manager), the degree 
f senior management involvement in evaluations (e.g. 
dvisory committees), the packaging of the issues, and 
he availability of suitable resources. 


here are many analytic techniques of varying cost and 
ertinence for evaluation in this area. Expertise in these 
echniques is not distributed uniformly among evaluation 
roups. Full-blown benefit-cost analysis, with monetary 
aluation of the benefits of alternatives (either ex post or 
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ex ante), is one of the most costly and least useful analytic 
approaches to alternatives. In addition, benefit-cost anal- 
ysis is strewn, with numerous pitfalls, and sometimes 
yields conclusions which do not hold up when subjected 
to a sensitivity analysis that tests the effect of varying 
the assumptions used. # 


5. Objectivity and Role 
of Line Management 
in Evaluation 


Issues 


Several issues were proposed for discussion: 


e For line managers, is program evaluation seen as 
“part of the solution”, for purposes of management? 
accountability? 


¢ To what extent is “program accountability planning” 
a help? 


¢ How can line “ownership” be provided? 


Discussion 


Line managers have both positive and negative perspec- 
tives on evaluation. On the positive side, evaluation justi- 
fies resources or products, makes a case against cuts, 
eliminates unwanted programs, helps allocate resources 
and helps establish new projects. On the other hand, 
evaluation also identifies deficiencies, supports program 
cuts, increases demand for program services and chal- 
lenges line information. 


A number of lessons have been learned regarding line 
management’s role in evaluation: 


a) Committees established for evaluations should 
include line managers, informal communications 
between evaluators and line managers should be 
improved, and line managers should be more closely 
involved in evaluation planning and linking to 
strategic planning. 


b) Evaluators are more frequently being asked for 
help on special studies, more, however, for their 
expertise than for their independence and discipline. 
Nevertheless this raises the profile of evaluation. 


c) Agency corporate priorities are more evident and now 
focus more on results and increased negotiation with 
line managers. 
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d) Evaluators now receive better feedback from line 
managers and tend to provide a united front to senior 
management. 


e) Evaluators have a better understanding of deputy 
ministers and government priorities. Their tech- 
niques are becoming more acceptable. 


Suggestions 


e Line management understanding and acceptance of 
evaluation is improving and techniques to obtain 
that acceptance are well understood. However, the 
need to retain control of the evaluation process in an 
“independent” evaluation unit is still extremely 
important. 


e Widespread use of evaluators’ skills in special studies 
that are evaluations in all but name would seem 
to indicate either a desire not to accept the evalua- 
tion discipline, or not to be associated with the 
government-wide system which leads to central 
documentation and perhaps easier public access to 
study findings. 


e Within departments, only deputy ministers and cor- 
porate interests are likely to value the credibility and 
accountability benefits of formal evaluation. m 


6. Program Evaluation and 
Internal Audit 


Issue 


The issue for discussion was the relationship and distinc- 
tions between evaluation and audit in departments and 
whether there was a need for greater clarification. 


Discussion 


Line managers generally do not distinguish sharply 
between departmental internal audit and evaluation, 
although to the extent that the distinction is made they 
tend to accept “white hat” evaluation better than “black 
hat” audit. Evaluation reports tend to be seen by line 
managers as forward-looking and of use in improving 
programs, while audit is not seen by managers as serving 
this purpose but as being fairly useful for detecting cer- 
tain operational deficiencies or risks. Directors generally 
agree that managers should be aware of the distinctions. 


While both functions assess broadly defined efficie 
and effectiveness issues, audit focuses inward on dep! 
mental operations — particularly, management practi 
procedures and controls — while evaluation focuses 
results and program performance beyond the depart 
— on program clients and broader socio-economic impal 
The issues addressed in evaluations are influenced he 
ily by managers and other stakeholders, while the iss 
addressed in audits are mainly chosen by the audit 
Evaluations tend to develop more distinct lines 
evidence and to take longer than audits. 


These distinctions may be becoming blurred as audit 
move towards results-driven audits. Evaluation di 
tors (and senior managers and the Office of the Aud 
General) are concerned that internal audit fulfil its ¢ 
mandate to attest to the soundness of managerial st 
ardship (prudence, probity, etc.) of resources. : 


Evaluators and auditors have different skill sets and | 
tural background. Persons in one group could effectix 
be assigned to conduct the other type of study only if 
task were simple or after significant training. | 
Frequently, evaluation and audit are “twinned”, but 
fused or integrated, and report administratively to 
same senior manager (generally the assistant dep 
minister, sometimes the director general). In small ag 
cies, the evaluation and audit function frequently « 
sists of one director and 2-8 officers for both together. 


Suggestions | 
In large and medium-size departments, an audit and 
evaluation of the same program are sometimes plan: 
concurrently but are conducted and reported separat 
There is no general rule in this area, however. A perc 
tion by managers of a merger or a close relations 
between audit and evaluation was viewed as reduc 
evaluation’s role in policy matters and in making eval 
tion appear more as a control function. 


Audit and evaluation seem to communicate more thar 
the past and enjoy a good working relationship (or € 
simply ignore one another). No relationship was repor 
to be bad. Mr. Doug Wood, Acting Deputy Comptro 
General, Audit and Review Branch, OCG, reported t! 
some audit directors view with concern evaluatic 
interest (and potential encroachment) in administrat 
policy matters. 


Generally, there was not considered to be much 0 
problem. = 
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7. Barriers to More Productive 
and More Effective Evaluation 


Issue 


These syndicate sessions considered what barriers, if any, 
existed to more productive and more effective evaluation, 
what could be done to reduce these barriers and who 
should take the initiative. 


Discussion 


While there was considerable debate about how to mea- 
sure an evaluation function’s productiveness and effec- 
tiveness, there was general agreement that there are a 
number of barriers that keep evaluation functions in the 
federal domain from being more productive and more 
effective. A number of these were identified: resource 
constraints (not only the quantity, but the quality of 
resources available); political factors (i.e. dealing with 
stakeholders both inside and outside the department or 
government); ad hoc demands (which cause a disparity 
between evaluation plans and performance by distracting 
resources from their planned purpose); departmental 
attitudes (acceptance, confusion between audit and eval- 
uation, access to deputy head, etc.); consultants (boon or 
bane?); institutional constraints (such as the Privacy Act, 
survey reviewing, staffing and contracting processes, 
etc.); and the quality of OCG liaison. 


Suggestions 


Some solutions were suggested: increased consultation, 
and a campaign of education and advocacy (to address 
political and attitudinal factors); selective focus for evalu- 
ation studies, and the use of co-op and summer students 
(to address resource constraints); selective use of consul- 
tants (for specific tasks, excluding final report); training 
(to address quality of resources); and the use of standing 
offers (to expedite contracting). 


The size of the evaluation unit and its location in the 
department, and whether the director of evaluation sits 
on a Senior Management Committee were not thought to 
be problems or solutions. m 


ormation please ties Judith Légaré or 
pois Lapointe at (514) 682-3400. 
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8. Program Evaluation and Policy 
Development 


Issue 


The issue discussed was the role of program evaluation in 
policy formulation and change in departments, and the 
factors which impede the effective linkage between policy 
and evaluation. 


Discussion 


It was quite evident that the directors at the two syndi- 
cate sessions were very interested in the subject of link- 
ing evaluation to policy development and change. This 
interest was apparent despite the current government 
preoccupation with downsizing and deficit reduction, 
which often shifts the focus of evaluations away from the 
policy concerns. 


Suggestions 


The following suggestions for furthering the role of a pro- 
gram evaluation group in policy-related initiatives 
emerged at the syndicates: 
e Senior managers should be further sensitized to the 
role of program evaluation in the policy formulation 
and change processes. 
° Careful decisions have to be made on timing and on 
subject-areas in which program evaluation should 
intervene. This is essential, as the locus of policy for- 
mulation is often diffused and there is a danger of 
diverting scarce evaluation resources without due 
return. It was a shared perception that the deputy 
minister’s key initiative area would be a likely tar- 
get, but to play an active role in such an area would 
require the confidence of the deputy that the evalua- 
tion group could do the job. 
e The neutrality of the program evaluation function 
must be guarded in the policy-making process. It is 
possible that even a perception of partiality could 
compromise the credibility of the function. 
¢ Though there are a number of management and 
resourcing issues, an interdepartmental evaluation 
could be an effective way to treat a key government- 
wide policy issue. In this regard the OCG’s role would 
be significant 
e Evaluators might take the initiative by 
— summarizing the evaluation findings from sev- 
eral evaluations in a policy area 

— providing a challenge to the policy function by 
making available pertinent evidence on past pro- 
gramming efforts when new policy proposals are 
being disucssed. m 
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GPEP Update 


The OCG is preparing the 90/91 version of the 
Government Program Evaluation Plan (GPEP), aiming 
for delivery this fall. As with last year’s plan, we expect 
to provide a brief overview of recent developments that 
might affect program evaluation, such as PS 2000, and 
the status of the function across departments. We also 
will update developments in various sectors and pursue 
the cost-effectiveness theme introduced last year. 


In addition, we are planning a chapter on improving the 
“productivity” or “value-added” of program evaluation 
functions in departments. We believe that there is sub- 
stantial scope to increase the utility of program evalua- 
tion in many departments and agencies as well as its 
worth in supporting interdepartmental government- 
wide initiatives. This chapter will focus on some of the 
barriers to more productive evaluation and include 
suggestions and initiatives for overcoming them. m 
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Our Community 


The program evaluation community 
is small relative to most other func- 
tional communities such as person- 
nel, finance and audit. As many eval- 
uation units consist of only a few 
people, good communications within 
the community are important. 
The evaluation function attracts peo- 
ple who will stay for shorter as well 
as longer periods of time. Some peo- 
ple work in evaluation units for a 
short specific length of time; for 
example, to provide subject-matter 
expertise to a particular project 
team. Others choose to spend a 
longer period of their Public Service 
career working as evaluators. Thus, 
mobility in and out of, and within 
the community is reasonably high. It 
is important that evaluators get to 
know each other, what we can 
expect, and what is expected of us. 


4 | Treasury Board of Canada Conseil du Trésor du Canada 
Comptroller General 


Program 
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Program Evaluation Branch 


Guy Leclerc, Deputy Comptroller 
General, Program Evaluation, 
recently invited new entrants to 
the community to attend the 
first “Welcome” session. This event 
was held on the morning of 
Thursday, November 8, 1990. 
Mr. J.A. Macdonald, the Comp- 
troller General of Canada, opened 
the session. Guy Leclerc; John 
Mayne, Director, Policy Division, 
OCG; and Martin Ulrich, Director, 
Operations Division, OCG were the 
key “hosts”; they provided an 
overview of current issues. Hervé 
Déry, a representative of the 
Canadian Evaluation Society (and 
evaluation analyst with the OCG), 
was also on hand. The guests had 
the opportunity to meet and speak 
with other members of the Program 
Evaluation Branch. 


ivi 
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A second session was held on January 4. Both groups 
were quite representative of the community as a whole: 
each contained junior- and senior-level people, full-time 
evaluators as well as Co-op students. 


We will continue to use our information bank to identify 
new members of the community to be invited to future 
“Welcome” sessions. All officers who currently occupy a 
program evaluation position in an organization of the 
Federal Government should be included. 


We look forward to receiving suggestions as to how 
these sessions can be made more valuable as well as 
suggestions and continued support for initiatives to 
make the evaluation community a good place in which 
to work. = 


Rob Jones 


Dr. Jones is Senior Policy Analyst with the Program 
Evaluation Branch, OCG. 


Budgeting, 

Program Performance, 
Program Evaluation: 
The MYOP Process 


The relationship between departmental program evalu- 
ation and central budget determination processes has 
long been seen as tenuous. The recent Treasury Board 
Secretariat (TBS) review of departmental Multi-Year 
Operational Plans (MYOP) was quite explicit in its 
review of program results performance. The review, of 
course, dealt with other factors, some related, such as 
budget determination, assuring consistency with pre- 
vious decisions, inflation adjustments, tax change 
effects, and person-year strategies. 


In global terms, the objective of the MYOP review pro- 
cess is to allocate resources to approved programs in 
response to government priorities and within fiscal pol- 
icy constraints and to do so in a way that encourages 
departments to do the same for their internal alloca- 
tions among activities and organizations. 


There are several means available to give effect to 
discussions and decisions, including: 


a) setting or recommending resource reference le 
controlling certain allotments of funds pen 
further information, and establishing cost-reco: 
targets; 


b) identifying potential Special Operating Agen 
adjusting the balance between making or by 
certain goods and services, and considering ot 
organizational actions; 


c) identifying issues (including program performe 
and program evaluation issues) for consultati 
with departments; 


d) identifying issues for consideration by 
Ministers, such as the need for review of spe 
programs, broader interdepartmental concerns 
references to other Cabinet committees. 


The MYOP review is much more than a process for « 
sidering marginal departmental resource reque 
Where marginal resourcing is proposed, the basis 
TBS recommendation is consideration of the efficie 
or cost-effectiveness of the delivery of the entire ] 
gram/activity. Where there is little evidence of efficie 
or use of appropriate cost-recovery, there is little bs 
for justifiying additional resourcing to the Treas 
Board. 


Cost-effectiveness is not an exact science. In additio1 
consideration of findings from evaluation, a numbe 
other factors play a role. Clear increases in workl 
without resourcing increases in the recent past, & 
maintenance of service levels with a declining resou 
base, are important indicators. In addition, a dep: 
mental approach to program management, which ir 
cates shifts of resources to evidently higher priorit 
and use of better targetting and cost-recovery g 
futher evidence of cost-effectineness. 


The key point, however, is that considerations of p 
gram performance do play a significant role in det 
mining resourcing and other kinds of actions no’ 
above. Cases where there was good information on! 
continuing need and cost-effectiveness, combined w 
evidence of management responding to this inforn 
tion, seemed the more successful. Evaluation findin 
where relevant and available, were considered. 


| 
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Of course, for most programs, recent evaluation find- 
ings were not available. This is due to the periodic 
nature of most evaluation as well as coverage of pro- 
grams well below Treasury Board expectations in 
many departments. 


From the perspective of the program evaluation func- 
tion, a concern greater than inadequate coverage is 
the relatively high number of evaluations that were of 
only limited help. Since departments are facing many 
of the same pressures as the Treasury Board, one 
would expect such gaps in issue coverage to limit the 
usefulness of evaluation for departmental managers. 
It is important to stress the substantial differences 
among departments in this regard. 


The principal weaknesses were generally those identi- 
fied in the 1989 Government Program Evaluation 
Plan, namely the absence in many evaluations of a 
rigorous analysis of continuing social and economic 
need for federal resourcing, and a seemingly overcau- 
tious approach to consideration of alternatives, partic- 
olarly less costly alternatives. 


Satisfied clients and improvements in delivery proce- 
dures, for example, seem to be addressed more fre- 
quently than the continuing social and economic need 
for programs. Many evaluations do examine need 
quite rigorously. However, many others do not or base 
their argument on mandate and mission, sometimes 
providing more conclusions than findings. 


With questions such as creation of special operating 
agencies, greater efficiency, and cost-recovery facing 
departments, alternative approaches to delivery are 
central issues. Cost-recovery is particularly important 
because it contributes to improved equity and 
improved targetting of program services, as well as 
reducing pressure on the deficit. Since program evalu- 
ation is better suited than other performance mea- 
surement instruments to address cost-recovery, alter- 
natives and cost-effectiveness questions, this 
weakness in many evaluations is particularly notice- 


able. 


An important factor in raising acceptance of depart- 
mental program evaluation is reporting which distin- 
zuishes clearly between findings and conclusions. 
Many evaluation summaries (and departmental Part 
(Is) focus more on conclusions than findings. A report 
mM performance which states the conclusions of a 
Jepartment or an evaluation study without reporting 
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the salient evidence or findings to support that conclu- 
sion carries little weight. 


In the current environment program evaluation can be 
quite instrumental in resource allocation, including the 
resourcing activities of the Treasury Board. But, to be 
useful, it is important that certain issues be addressed 
rigorously and findings be reported clearly. More evalu- 
ations are now meeting these high expectations, but 
many more such evaluations are needed. = 


Martin Ulrich 


Dr. Ulrich is Director of the Operations Division, 
Program Evaluation Branch, OCG. 


. The theme will be “The Politics of Evaluation The - 

_ objective is to explore the political influences 
_ which are inherent i in any evaluation research pro- 
cess and often affect evaluation design, implemen- | 
- tation and utilization. The challenge is to under- 
stand more fully the implications of the increased — 
| political sophistication. For: more et ota you . 
may contact: _. _ 


_ nent Becnsiatiat 

co Events By Design 

Suite 601, 325 Howe Street 

| Vancouver, British Columbia 
a V6C A oe 
Tel. (604) 669- 7175 
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New Directors/Heads of Evaluation 


Since September 1, 1990 the following people have assumed the duties of Director/Head of evaluation: 


Angus, Craig 


Couturier, Pierre 


Holloway, Beverley 


Lahey, Robert 


MacDonald, Malcolm 


Macmillan, Allan 


McQueen, Cameron 


Maddocks, Roy 


Martin, Michael 


Tardif, Gilles 


A/Director 


Director 


Director 


Director 


Director 


Director 


A/Director 


Director 


Director 


Director 


Program Evaluation Division 


Canada Mortgage and Housing Corporation 


Evaluation, Audit and Statistics 
National Parole Board 


Program Evaluation 
Health & Welfare 


Evaluation 
Forestry Canada 


Audit and Evaluation 
External Affairs & International Trade 


Evaluation Branch 
Supply & Services Canada 


Program Evaluation Division 
Revenue Canada — Customs & Excise 


Program Evaluation & Research 
National Archives of Canada 


Corporate Planning & Coordination 
Emergency Preparedness Canada 


Financial, Planning & Administration 
National Transportation Agency 


Das you re an one article « on à program | | 
“evaluation that we can publish? - The Newsletter 
welcomes all contributions in either official _ 
- language from the evaluation community. For. : 
further information, please call the ee 


| Paulette Somerville . Via 957-71 67. 


The Evaluation of Public Policies 
in France 


In 1988, in order to meet the challenges of the European 
common market of 1992, and in particular the competi- 
tiveness of national governments, Prime Minister 
Rocard introduced a comprehensive plan for the mod- 
ernization of the French Public Service. In spite of sig- 
nificant differences between the Canadian and French 
situations, the similarities between the objectives of the 
French program of reform and the Canadian “Public 
Service 2000” project are evident: decentralization of 
management responsibilities, desire to encourage inno- 
vation at all levels, and especially at the lower levels, 
emphasis on the quality of service to the public, accent 
on results, etc. 


Within the evaluation community in Canada, it is gen- 
erally agreed that such reform cannot be achieved with- 
out an increased demand for information on program 
performance and consequently, an increased demand for 
program evaluation. This opinion tends to be confirmed 
by the French example where program evaluation is 
clearly seen as one of the key elements in the “renou- 
veau de |’Administration” (Public Service renewal) 
project. 


it should be pointed out, however, that until now the 
Government of France has not found it necessary to 
institutionalize a central evaluation function, in spite of 
marked interest in the discipline, especially from the 
university communities, and the example of a number 
f western countries, including Canada and the United 
States. Until very recently, the French government with 
its strong legalist tradition, has focused on compliance 
controls at the expense of monitoring the effectiveness 
of government actions. The recent attention given to 
evaluation is aimed at reestablishing a balance. 


Definition of the concept of evaluation: 
a difficult step but a necessary one prior to its 
institutionalization. 


Evaluation can mean different things and have 
i variety of objectives. Within the federal Public Service 
n Canada, for example, it is seen primarily as a 
Somponent of the departmental management process. 
According to Treasury Board policy, program evaluation 
nvolves both an empirical study of a program’s 
erformance and a review of the program and its 
erformance: specifically, its relevance, success 


EE 
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and cost-effectiveness!. This two-pronged definition 
acknowledges the two components of evaluation: the 
more objective or descriptive element involving the 
measurement of performance by an empirical study, and 
the more subjective or analytical element of making a 
judgement on the program’s performance through its 
review. 


It is often impossible to avoid a confrontation between 
the opposing concepts when deciding on a definition. 
French authorities were unable to escape it. The Prime 
Minister was given two reports on evaluation prior to 
the adoption of the decree of January 22, 1990 which 
institutionalized program evaluation. The reports 
offered very different definitions of the evaluation 
function. 


The first of these reports appeared in 1986 and stated 
that evaluation of a policy consists in identifying and 
measuring its specific effects?. This is a very general 
and academic definition: it presents a methodical 
approach which clearly shows the authors’ intent to 
relate evaluation to a scientific discipline. This vision of 
evaluation is drawn to a large extent from American 
practice in the late 60s and early 70s when extremely 
ambitious studies such as the “New Jersey Negative 
Income Tax” were launched. This definition of the 
function focuses solely on the descriptive aspect of eval- 
uation and makes it a primarily cognitive function; it is 
seen mainly as a means of increasing awareness of gov- 
ernment programs and policies by updating their 
impact. Because it does not include the more analytical 
and subjective aspect of the evaluation function, that is, 
the idea of judging the performance of a program, this 
definition is quite removed from Canadian practice. 


The author of the second report (published in 1988), 
while criticising the “positivist” and “technocratic” con- 
cept of evaluation in the first report, promoted a more 
flexible definition that focused more on the democratic 
nature of evaluation. He therefore suggested that evalu- 
ation of any public action consists in rendering a judge- 
ment on the value of that action; he went on to say that 
in a democratic system, only elected officials and courts 
have the legitimacy needed to render such a judge- 
ment‘. Contrary to the approach of his predecessors, 
the author of this report dealt solely with the more sub- 
jective nature of evaluation. Evaluation is no longer 
regarded as an empirical instrument of measure but 
rather as a value judgement. In this regard, the defini- 
tion offered in this report is more ambitious than that in 
practice in Canada. The Canadian approach is that 
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evaluation should include an assessment of the perfor- 
mance of a program, including its relevance, cost- 
effectiveness and ability to meet its objectives. This is 
quite different from an assessment of the value of a pub- 
lic activity or program. The concept of value is broader 
and less restrictive than that of performance. For exam- 
ple, a program’s performance can be judged to be poor in 
terms of effectiveness and administrative efficiency and 
yet still be of important value to political leaders and 
the public. Conversely, a program’s performance could 
be judged outstanding without political leaders or the 
public considering it to have much value. It is therefore 
easy to understand why the author of this report saw 
elected officials as the main clients of the evaluation 
function. 


The definition ultimately used in the January 1990 
decree undoubtedly stems from a desire to compromise 
and a refusal to categorically rule in favour of one or the 
other of the proposed definitions. The order states that 
evaluation of public policies involves measuring the 
effects that they have and determining whether the legal, 
administrative and financial means used accomplish 
the desired effects®. This definition incorporates both the 
descriptive and analytical aspects of evaluation. 
Although the first part reiterates almost verbatim the 
definition used in the first report, the second part falls 
well short of that given in the 1988 report, or indeed of 
what is actually practiced in the Canadian federal 
Public Service. There is no mention of judging the value 
or performance of a policy or program, but essentially of 
assessing the effectiveness of the means used to achieve 
the objectives. To Canadian evaluators, this may appear 
to be a somewhat restrictive definition. There will be a 
tendency to believe that the emphasis placed on the 
assessment of the effectiveness of the means used could 
overshadow other issues considered fundamental in the 
Canadian evaluation context, such as: what is the pur- 
pose of the policy or project? Does the program still 
reflect government priorities and current needs? Does 
realization of the objectives give rise to excessive costs 
or “secondary” effects? etc. An examination of how the 
French put this definition into practice may enable us to 
determine more precisely the degree to which this spe- 
cific definition of evaluation determines actual practice 
and sets French practice apart from that seen in the 
Canadian government. 


Institutional organization of evaluation: 
centralism and domination of the Executive 


Officially, the French government wants evaluation 
have a two-fold objective, as was evident from { 
speech of President Mitterrand given in July 19} 
(1) advancement of democracy, considering evaluat 
like an instrument of political accountability, a 
(2) improved performance of State activities, as a co 
ponent of the management process®. To achieve tl 
end, a three-tiered system was introduced: 


(1) Comité interministériel de l'évaluation (interdepa 
mental committee on evaluation): Chaired by t 
Prime Minister, this committee is responsible : 
developing and coordinating government evaluati 

initiatives. It orders interdepartmental evaluati 

projects, as well as projects using a fund specifica 
for evaluation studies (Fonds national de dévelop 
ment de |’évaluation) (national evaluation devel 

ment fund). Suggestions for evaluation studies c 

be submitted to the committee by various admin 

trative audit, monitoring or socio-economic planni: 
organizations. The committee is also responsible { 
approving the modalities of studies, in particul: 
the criteria for the selection of the individuals, pu 
lic or private, who will conduct the evaluation. 


(2) Fonds national de développement de l’évaluatio 
The funds in this account are used to carry 0 
evaluations approved by the interdepartment 


committee. | 


(3) Conseil scientifique de l'évaluation (scientific eval 
ation board): This organization consists of eleve 
individuals appointed by the Prime Minister to 
six-year non-renewable term. The members a 
selected on the basis of their expertise in the field! 
evaluation, economics, sociology or administratio: 
Their mandate is to promote the development 
evaluation methods, to define a code of ethics for th 
evaluation profession, and to ensure the quality an 
objectivity of studies approved by the interdepar 
mental committee. The Conseil is therefore require 
to provide advice on the methodology to be used i 
such studies, to evaluate the quality of evaluatio 
studies, and to assist in the training of specialists 0 
the development of research in the evaluation fielc 
It reports on the fulfillment of its mandate in a 
annual report describing the evolution in evaluatio: 
practices. 


—— 


keeping with French government tradition, the orga- 
zation of the evaluation function is very centralized. 
Departments and agencies have no formal authority in 
he field. France is not unique in the centralization of 
aluation: one only has to think of the General 
counting Office in the United States. On the other 
nd, the French have also set up a system that is con- 
tolled almost entirely by the Executive, as is the case 
in the Canadian government. This structure faithfully 
reflects the organization of the political institutions of 
the Fifth Republic where, as in the British parliamen- 
system, the Executive largely controls the legisla- 
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the approach adopted by France is innovative in a 
mber of ways. Firstly, members of the government, 
d primarily the Prime Minister, are responsible for 
the direction and coordination of the evaluation func- 
ion. Secondly, an independent administrative body (the 
Conseil scientifique) has been created to ensure the 
bjectivity of the function by providing advice on 
nethodology and by monitoring the quality of the work. 
Thirdly, the institutional structure of the evaluation 
unction clearly shows that its implicit objective is to 
einforce the political control of the government over 
he administration. By making the Prime Minister and 
is colleagues the main clients of the function, the aim 
s to provide the government’s decision-makers with 
bjective information on programs which they cannot 
btain from the bureaucracy, and which will give them 
lew input when making decisions. In other words, the 
valuation function becomes merely counter-expertise 
0 the knowledge of the bureaucracy. In that way, it will 
lelp reestablish ministerial authority in decision- 
naking by undermining the bureaucracy’s monopoly on 
mowledge of program operations and performance. 


n conclusion... 


the decree that institutionalized program evaluation in 
'rance has scarcely been in force a year. It is therefore 
00 early to report definitively on the French experience. 
Ne will simply identify some of the positive and nega- 
ive factors that may affect the future of evaluation in 


‘rance. 


'irst, we feel there are three factors that may impede 
he development of a healthy evaluation function in 
rance. The first is the weight of bureaucratic tradi- 
ions. The elite of the French Public Service, who are 
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part of the government’s “Grands Corps”, appear not to 
have been involved in the evaluation reform. This may 
be worrisome. Many studies have identified the consid- 
erable power of the “Grands Corps” and the need for any 
government to have their support if it wishes to ensure 
the success of its reform projects’. Organizations such 
as the “Inspection des Finances” or the “Cour des 
comptes” may well view evaluation as a competitor and 
perhaps wish to derail it. 


The second factor is the need to anticipate possible 
favouritism in the awarding of contracts for the actual 
realization of the evaluations. It should be remembered 
that it is members of the government — those sitting on 
the interdepartmental evaluation committee — who are 
responsible for setting the standards and selection crite- 
ria for contractors. Of course, the “Conseil scientifique”, 
an independent body, will be evaluating the evaluation 
work performed, which should ensure the selection of 
qualified contractors capable of providing credible find- 
ings that are sufficiently objective. This does not how- 
ever eliminate all possibility of favouritism coming into 
play to the detriment of the quality of evaluation 
studies. 


Thirdly, there is the question of whether the political 
control exercised over the evaluation function through 
the interdepartmental committee limits the critical 
aspect of this function. Since it is members of the gov- 
ernment who determine which programs or policies are 
evaluated, it is unlikely they will want to cut their own 
throats. Consequently, the policies subjected to evalua- 
tion may not be those that are most politically sensitive 
or controversial. This point has already been raised by 
critics, 


Nevertheless there are also a number of factors that are 
favourable to the development of the evaluation func- 
tion in France. France will be able to draw on the expe- 
rience of other countries to avoid several of the pitfalls 
that inevitably befall new initiatives. Evaluation cur- 
rently enjoys an uncommonly large degree of political 
support among the highest levels of government. And 
lastly, it is being implemented as part of a much broader 
program of reform within the Public Service, which 
should help create a more favourable climate for 
change. We therefore hope that this climate, conducive 
to the establishment of the evaluation function, endures 
and allows the Government of France to assume a posi- 
tion as a world leader in the evaluation field. 
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Treasury Board of Canada, Program Evaluation 
Policy 


Deleau M., Nioche J.P., Penz P. and Poinsard R, 
Evaluer i politiques bone méthodes, don: 
tologie, organisation (evaluating public policies: 
methods, ethics and organization), Paris, La 
Documentation française, 1986. 


This refers to the famous “ex-ante” inquiry aimed at 
measuring the effects of subsidies to underprivi- 
leged families on their socio-economic behaviour. 
Several authors have referred to this classic case. 
For a summary of the study, see: Kershaw, D.N., 
“The New Jersey Negative Income Tax Experiment: 
a summary of the design, operations and results of 
the first large scale social science experiment” in 
Lyons G.M. (ed), Social Research and Public 
Policies, OECD Darmouth Conference, Darmouth, 
1975, pp 87-116. 


Viveret P., L'évaluation des politiques et des actions 
publiques (evaluation of public policies and actions), 
Paris, La Documentation francaise, 1989. 


Decree No. 90-82 of January 22, 1990 on the evalua- 
tion of public policies, Paris, Journal officiel de la 
République française, January 24, 1990, pp 951-3. 


On this topic, see: the speech of President Mitterrand 
at the creation of the Conseil scientifique d’évalua- 
tion des politiques publiques, Paris, July 13, 1990. 


The power of the “Grands Corps de l’État” (senior 
government officials) has been studied, among 
others, by: Marie-Christine Kessler in Les Grands 
Corps de l’État, Paris, FNSP, 1986, and by 
Catherine Grémion in Profession: décideur, pouvoir 
des hauts fonctionnaires et réforme de l’État, Paris, 
Gauthier-Villars, 1984. 


See for example: Nioche J.P., Evaluation in France, 
a paper presented at the international conference on 
program and policy evaluation, Hague, December 
1990. 

Marc Lemay 


Mr. Lemay is a research analyst with the Program 
Evaluation Branch, OCG. 


Canadian Evaluation 
society 

in the 1990s : 
Evolving with 
the Times 


Fifth Annual One Day Workshop 
November 14, 1990 


“Evolving with the times” appears to be a classic unde 
statement; if I had to find one word to describe the pr 
sent evaluation environment and the CES event, th 
the word would be change. The theme of ongoit 
change was certainly pervasive throughout the day. / 
a newcomer to the OCG, this was reassuring - it mear 
that others do not have all the answers and are al: 
having to adapt to a new environment and differel 
ways of doing things. 


So what are all these changes? Instead of a chronolog 
cal “resumé” of the day, I will attempt to weave mn 
notes under three inter-related headings: | 


1. Changes in the role of government in society; 


2. Changes inside government — new ways of doin 
things; and 


3. Changes in evaluation as a professional disc 
pline. 


1. Changes in the Role of Government 


As luncheon speaker, Nicole Morgan (author « 
Implosion and The Equality Game) provided som 
thought provoking topics: 


° The government is having a mid-life crisis 
Originally bureaucracy was developed as an alterna 
tive to survival of the fittest (she also referred to thi 
concept as the “bully syndrome”) which in economi 
terms translates into market forces. Government 
embodied in the bureaucracy, evolved as an alterna 
tive force to protect the vulnerable in the ongoins 
domestic and global economic wars. 
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But in an information society, BIG bureaucracy often 
cannot react in a timely fashion and some govern- 
ments are returning to the market forces concept as 
perhaps a better way of doing things. And she was 
quite critical of this regression to the “bully 
syndrome” and suggested that the Public Service 
should address what is the proper balance between 
bureaucracy or public good and the ‘privatization’ 
of government. 


¢ There is an ongoing ‘politicization’ of the 
Public Service. She stressed that the Public 
Service mirrors national and global trends and that 
there is a need for a better balance between “image” 
(now an unfortunate but crucial dimension in poli- 
tics) and substance. She expressed grave concerns 
about bottom line, i.e. “looking good” in the bureau- 
cracy and the obsession of packaging and selling your 
‘product’ in the system. The victim of this process is 
genuine and honest research which is falling to the 
wayside. Right now she sees that the idea or pro- 
gram with the most political clout (the bully) pre- 
vails. 


With respect to some of the more political issues, (e.g. 
women’s or francophone representation), she pre- 
sented the idea that establishing official ‘minorities’ 
in fact creates new ideological ghettos which the per- 
son who is labelled as a minority can no longer 
escape from. This was also given as an explanation of 
why certain groups of Canadians, such as natives, 
refuse to self-identify. The alternative to the 
majority-minority polarization was suggested: we 
need to move away from looking good (i.e. doing 
something about the issue in a superficial way) and 
instead carry out rigorous research and seriously 
think through what kind of multi-cultural or egali- 
tarian Canadian society we want. 


* Where do we go from here? Lastly, she left us 
with an important question about Access to 
Information, a freedom which carries a price: the 
potential negative impact that it may have on honest 
(where things don’t look good) research. There was a 
Suggestion that perhaps certain research studies 
Should be protected within the government to 
encourage a more substantive discussion of the 
issues and statistics. Ultimately this solid research 
Should determine what kind of Public Service we 
should have in 2001 and how we should be working 
towards it. 
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2. Changes Inside Government 


The keynote address by Timothy Plumptre focused on 
this topic within the context of PS2000 and the Service 
to the Public Task Force report in particular. He raised 
the paradox that faces the public service — the general 
public’s view of bureaucracy as slow and unresponsive, 
yet high expectations that bureaucrats will not only 
solve complex problems but do so in a climate of fiscal 
constraint. Given that the above report has received 
wide circulation I will not undertake a précis. Suffice it 
to say that the bottom line is a ‘best practices’ approach 
using successful organizations as models, that is, the 
answer of the task force is to give good service to the 
public and place a high value on good management 
of human resources... an easy to say, hard to do 
initiative. 


3. Changes in Evaluation 


Timothy Plumptre did, however, go on to expand on the 
implications of a “Service to the Public” thrust on the 
evaluation community, given that government has regu- 
latory or unpopular functions. The job of some programs 
is to have unhappy clients (e.g. catching law breakers). 
And government is more complicated than the largest 
private organization, Public Service employees must 
often manage programs that have multiple and perhaps 
incompatible objectives that span the socio-economic 
dimension. Futhermore, as Toru Uno of the Program 
Evaluation Branch of the OCG also noted, increased 
accountability, coupled with deregulation and decentral- 
ization trends, requires very sophisticated handling: 
how to introduce some common standards for perfor- 
mance (especially in the service to the public sphere) 
without over-designing government organizations and 
processes. 


Despite these challenges, throughout the conference an 
overall concensus emerged on the evolution of evalua- 
tion, that it should be more strategic and link more 
closely with decision-making as one form of service and 
support to management. This policy stance requires all 
of us to reexamine questions such as who is the primary 
client, what measures to monitor and how, and to 
demonstrate in actions that staff (e.g. evaluation 
groups) are valued as a corporate resource. 
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William (Bill) Smith of NRC carried on the theme of 
changes to evaluation in his closing remarks as he 
explored the following: 


¢ to link, where appropriate, evaluation closer to key 
issues and policy concerns (e.g. competitiveness and 
environment), as well as the more traditional links to 
operational management; 


e to continue to develop evaluation as a professional 
discipline and support the Canadian Evaluation 
Society as one instrument to achieve this; 


* to reach out and build a broader understanding of 
evaluation as one instrument in the corporate tool- 
box to support managers; and 


e that evaluation faces both challenges and exciting 
new opportunities in a climate of increasing account- 
ability and that this challenge will require all of us to 
be innovative. 


In addition to the above speakers, there were seven 
workshops. A listing of these follows with the names of 
the moderators* and speakers should you wish to follow 
up on any material: 


A. Evaluation Audit, Market Research: Partners for 
the Nineties: *Garry Sears, Peat Marwick 
Stevenson and Kellogg; Suzanne Lafortune, 
Young & Wiltshire and Carmen Young, NCC. 


B. The New Context of Evaluation: *Hervé Déry, 
OCG; Brian Emmett, EC; David Black, ARA 
Consulting Group and Guy Leclerc, OCG. 


C. Program Evaluation — New Horizons and 
Applications: *David Moynagh, CCAF; Robert 
Hanson, SSHRC; Ian Midgley, CEIC and Alan 
Winberg, TBS. 


D. Evolving Methods: *Steve Montague, Young & 
Wiltshire; Tina Beranbaum and Ellen Corkery- 
Dooher, Coopers & Lybrand; Mike Neals, Gallup 
Canada; George Teather, NRC and Keith 
Vodden, Abt Associates. 


E. Evolving Education: *Francine Rancourt, 
Université du Québec à Hull; Karl Boudreault, 
INAC; Rob Jones, OCG and Jean George 
Lengellé, Université du Québec à Hull. 


F. Evolving Applications: *Robert Letellier, Co 
John Fisher, TC; Jac van Beek, Young | 
Wiltshire and Shelley Borys, Com. 


G. Evolving Accountability/Clientele, An Interdep ! 
mental Evaluation, PERD: *Alain Barbarie, E 
Robert Cook, OCG; Langley Muir, EMR a 
Douglas Williams, DPA Group. 


In conclusion, I hope my rather amateur journalis 
does not in any way detract from the excellent organiz: 
tion of the event and the personal dedication that t 

CES ‘ringleaders’ provided to make the day a succes 
I am looking forward to next year’s session. In t 

meantime, if you have comments on the above or ar 
particular workshop, please send them in to the eit 
of this Newsletter and share your insights with tk 
rest of us. = 


| 
Nola Breithauj 


Nola Breithaupt is Chief, Science, Economic an 
Regional Development, Program Evaluation Brancei 
OCG. | 
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Newsletter 


Changes at the OCG 


This Spring, the Comptroller General 
announced the reorganization of the 
Progam Evaluation and Audit and 
Review Branches into a single branch. 
This was done in order to focus perfor- 
mance definition and measurement 
expertise for program, operational, 
and administrative performance in 
one branch in TBS\OCG. The reorga- 
nization not only saves some re- 
sources, it provides better cross- 
fertilization of ideas. It will make the 
OCG more effective in coordinating 
central demand and in having each of 
the review functions play their impor- 
tant role in accountability and cost- 
effectiveness. And it will help in mar- 
keting these services to deputies and 
senior and line managers. The impor- 
tant distinctions between audit and 
evaluation, reflecting the skills used 
and the issues addressed will, how- 
ever, be maintained. Doug Wood, who 
has been the acting Deputy Comp- 
troller General for the Audit and 
Review Branch, will lead the new 
branch. 
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Concurrent with these changes, Deputy 
Comptroller General Guy Leclerc, 
who led the federal program evalua- 
tion community for the past eight 
years, will be leaving the OCG for a 
new assignment with the Canadian 
Comprehensive Auditing Foundation. 


Leclerc’s significant contributions to 
the furthering of evaluation were 
recently recognized by the Canadian 
Evaluation Society (CES) when it pre- 
sented him with a Public Sector Award 
at the societys annual meeting in 
Vancouver. 


Over the years, Leclerc has made pre- 
sentations to both the CES and the 
American Evaluation Association. 
Last December he made a presenta- 
tion on program evaluation in 
Canada at an international confer- 
ence on the subject in The Hague, the 
Netherlands. 
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In announcing Leclerc’s move to the foundation, 
Comptroller General Andrew Macdonald stated that 
“Guy has been a valuable leader of the federal evaluation 
community . . . and his guidance in this area will 
be missed.” 


In his work with the auditing foundation, Leclerc will be 
responsible for drafting a major text on all the compo- 
nents of value-for-money audit, performance measure- 
ment, evaluation and management representations over 
the next couple of years. Comprehensive auditing has 
been around for the past 10 or 15 years and much expe- 
rience has been gained. It is therefore now an appropri- 
ate time to prepare a consolidated text describing and 
discussing the precepts and best practices of compre- 
hensive auditing, and associated value-for-money prac- 
tices. Leclerc has written on the subject and is looking 
forward with great enthusiasm to the work. And the 
work will keep him in contact with the evaluation 
community! a 


Exploring Expectations 
for Federal Evaluation 


One of the basic principles in program evaluation is that 
in order to be able to assess the performance of a program 
you have to know what is expected of it. As a result, con- 
siderable effort often is taken at the outset to develop a 
good understanding of and to get agreement by the 
affected stakeholders as to just what are the objectives of 
the program. And if reasonably clear objectives are not 
forthcoming, the evaluator resorts to a more goal-free 
approach reporting on what has happened as a result of 
the program rather than assessing it. 


It is unfortunate then that evaluators, and others who 
seek to assess program evaluation itself, pay little if any 
attention to discussing and explaining what they expect 
from evaluation. One must assume this is either sloppy 
research or it is believed that the expectations of evalu- 
ation are well known and accepted by most. This latter 
belief may be the case but it is clearly wrong. Even a 
casual look through the evaluation literature demon- 
strates the diversity of opinion as to what evaluation is 
all about. There is no accepted single purpose model of 
what evaluation is supposed to accomplish. The large lit- 
erature on evaluation utilization discusses numerous 
quite different types of utilization, suggesting a variety 
of possible aims for evaluation. Smith (1988) gives a 
recent review. 


| 
To take but one example, some authors seem to fee 
evaluation at the federal level is aimed or sho 
aimed at reducing the government's deficit (Hi 
1990; Savoie, 1990). Using this yardstick, it is re 
easy to draw firm conclusions about the efficacy of 
eral evaluation. But few involved with federal evalu: 
would subscribe to this purpose. And such assess 
do not discuss or explain in any detail just what i i 
intended direct connection between evaluation 
deficit reduction, or offer any evidence that this is in 
the avowed aim of evaluation. 


Treasury Board policy states that program evaluati 
meant to improve government programming, prc 
for accountability and improve policy developn 
but offers little guidance as to priorities among t 
general aims. So what are the expectations for fec 
evaluation? 


Following good evaluation practice, if the objective: 
not clear, go find out what actually is happening! To 
end, as part of a larger survey on evaluation utiliza 
more than a dozen of the more experienced federal d 
tors of evaluation were canvassed to see what they 
ceive as evaluation’s aim. They were asked to ran 
importance a number of possible purposes for evalu 
in their organization. This was followed by a discus 
of the question. 


The numerical rankings are summarized in the folloy 
table. 


‘Rae ‘of . 
the 4 respondent 
= : ~ porpg 


Evaluation purpose 


Corporate challenge/ 
second opinion 


Aid to strategic planning/ 
policy development 


Aid to program managers _ 
to improve delivery = eee 
A central Treasury Board ‘. me : 
requirement eas 


Means to account for program ce : 
performance | 


Means to identify dollar savings 
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Although many viewed evaluation primarily as a tool for 
assisting program managers to deliver their programs 
better, there was no general pattern among the respon- 
dents. The interesting finding is that, while corporate 
ederal evaluation practice across government falls 
mder the same umbrella Treasury Board policy and in 
ost cases can be said to emanate from that policy, 
deral evaluation plays different primary roles in 
lifferent departments. : 

as 

Through discussion it became clear that evaluation was 
indeed different in different departments and that the 
particular characteristic taken by evaluation in a depart- 
ment depended on a number of factors: 


the specific information needs of the organization, 

* the management style of the management executive, 
, the history of evaluation in the organization, 

. | the existence and role of other analytical groups, 


* the perception of evaluation held by the director of 
evaluation, and 


* the perception of evaluation held by the principle 
client for evaluation, the deputy head of the 
“organization. 


That no single government-wide purpose for evaluation 
existed was reinforced by several of the responses that 
sould not be recorded numerically. Several directors had 
great difficulty ranking the various purposes since, in 
their view, the primary purpose varied from evaluation 
evaluation based on the department's specific interest 
n the program at the time of the evaluation. In another 
ase, a primary purpose could be identified, but it had 
changed over time as the information needs of the 
Jepartment evolved. 


Thus, program evaluation responds to the perceived 
information needs of the department for program review 
md reconsideration. As management evolves in a 
lepartment, so does evaluation ifit is to remain effective. 
The broad-brush purpose given in TB policy documents 
brovides the flexibility for different types of evaluation to 
volve at different times and different places. 
Departments have various views on the primary purpose 
Mf evaluation, ranging from a control function which 
assesses program performance to departments that use 
valuation more as a form of management consulting 
0 line managers. This variety in purpose is consistent 
with Sonnichsen’s (1991) findings based on analysis of 
valuation practice in a number of U.S. government 
lepartments. 
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To some extent, departmental demand for evaluation is 
determined by factors outside of the control of the evalu- 
ation group: the nature of strategic planning in the 
department, the strength of other measurement sys- 
tems, the demands by Treasury Board and Cabinet, etc. 
But it was also clear that departmental demand for eval- 
uation was something over which evaluation, and in par- 
ticular the evaluation director, had some influence. Good 


marketing of evaluation and how evaluation findings can 
contribute to the organization, establishing the credibil- 
ity of the evaluation group and being involved in depart- 
mental strategic planning all contributed to a more 


significant and better-focused demand for evaluation. 
Expectations for evaluation can to a large extent be 


fostered and influenced. Mayne (1991) discusses this 
idea further. 


One can also approach the issue of expectations for eval- 
uation by looking less at the avowed purpose of evalua- 
tion and more at the actual impact of the evaluation 
function in a department. The Office of the Comptroller 
General has reviewed a sample of evaluation studies to 
get some idea of the types of program changes which 
have resulted from the evaluations. The resulting 
data were: 


Program termination 3% 
Significant program redesign 16% 
Program operational improvements 45% 
Confirmation/continuation of the program 8% 
Improved understanding of the program 26% 
No program use 2% 


In compiling this data, evaluations were categorized into 
their “most dramatic” utilization type, in the order 
shown. For example, “improved understanding” use 
implies no higher level use and operational improve- 
ments implies no major redesign of the program. 


There was general agreement among the respondents 
with this utilization data. As expected, there was indi- 
vidual variation among the departments, with the “con- 
trol” departments exhibiting relatively more confirma- 
tion.use and the “management consulting” departments 
claiming somewhat more redesign and operational 
improvement. 


The question was then discussed as to whether the data 
were consistent with expectations or represented a tar- 
get to aim for. However, there was general agreement 
that aiming at a target for utilization types has little 
meaning. This is consistent with the observation that dif- 
ferent departments have indeed different expectations 
and therefore no overall target for federal evaluation 
findings utilization made sense. 


Program Evaluation 


Furthermore, any such target or expectation would 
depend on just what one thinks of the state of govern- 
ment or departmental programs: if government pro- 
grams are thought to be generally quite deficient, one 
would expect quite different utilization of evaluation 
than if programs were thought to be generally all right. 
Similarly, if resource savings were critical, then one 
might expect more of that type of utilization. In a stable 
department whose programs were thought to be working 
well, evaluation might best play the role of challenging 
those assumptions, even if the typical result was recon- 
firmation of the program and/or modest operational 
improvements. 


The types of program impacts referred to above can be 
more or less measured, although attributing the change 
to evaluation can be problematic since program man- 
agers naturally wish to take credit themselves for 
improvements to programs. Nevertheless these impacts 
can for the most part be pointed at and might be thought 
of as a basis for expectations for evaluation. However, 
when asked to describe the major type of impact made on 
the organization, the majority of respondents did 
not refer back to these different types of program 
uses but, in fact, discussed aspects of a more general 
organizational use. 


That is, the major impact of evaluation in a department 
seemed to be not so much in the specific program 
improvements brought about — important though they 
were — but rather in the longer-term effect on the 
organization and how it manages generally. Example 
comments were: 


¢ long-term impact on the mix of programs and their 
designs 


* often a seed for new ideas which take some time to 
be accepted 


* improved understanding of what business the 
organization is in 


° better objective base of information available on 
program performance 


* opportunity to reconceptualize programs 


* opportunity to get good feedback with program 
clientele 


* accelerates program change in the organization 
° developed a corporate memory and perspective for 
the organization. 


On this basis, expectations for evaluation would involve 
something like improvements in results-oriented man- 


agement style in the organization, i.e. an improveme 
good management. 


Summary Findings 


So what are realistic and practical expectations for 
eral evaluation? At the aggregate level, they would] 
to entail a variety of aims to cover actual federal prac 
And they would have to include both program impact 
well as organizational impacts. 


The data suggest that 


successful evaluation over time should lead to 
a results-focused management style in an 
organization. 


Further, given the importance placed on improving 
delivery of programs and the historical data on u: 
evaluations to change programs, 


successful evaluation in the majority of cases 
should result in improved program design and 
delivery. 


This would include frequent cases of program resoui 
being reallocated within the program or elsewhere in 
organization. Finally, we might expect that 


successful evaluation will result in cases where the 
program is performing well and is confirmed and 
where the program is not working and is discontin 
ued, depending on the state of health of the organi- 
zation’s programs and the extent of the challenge 
role played by evaluation. 


Conclusions 


Are these expectations adequate? They are certainly c 
sistent with the Treasury Board policy on evaluati 
Are they consistent with the environment of the ’9 
With continued resource restraint, a move to empov 
managers and an increasing need to be able to dem 
strate performance, i.e. accountability, the governm 
is likely to expect that evaluation contribute to th 
ends. And indeed, the primary purposes identified e 
lier in the table fit quite well with such expectations: 


° Assisting managers to improve programming 
(including less costly program delivery alternatir 
and better strategic planning and policy), and 

¢ Enhancing accountability 


(demonstrating performance and challenging 
programs) 
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Furthermore, enhancing a results-oriented management 
style is exactly the kind of cultural change which the 
PS2000 initiative is looking for. 


These conclusions, of course, ignore the question, 
ow will these expectations be met?” It is here, perhaps, 
where more might be expected from federal evaluation: 
ore challenging evaluations providing relevant 
information to assist managers in making the tough 
resource reallocation decisions in this era of shrinking 
resources. m 
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Dr, Mayne is Director of the Evaluation Policy Division, 
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Do you have an interesting article on program eval- 
uation that we can publish? The Newsletter 
welcomes all contributions in either official 
language from the evaluation community. For 

her information, please call the Editor, 
| Paulette Somerville at (613) 957-7167. 
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Unlisted phone numbers: 
do they bias survey findings? 


The May 1991 Newsletter of the York University 
Institute for Social Research has an interesting note. On 
one of the Institute’s projects, a telephone poll was used. 
Instead of using the telephone book for the sample, num- 
bers were generated with random digit dialing so that 
every possible telephone number (listed or otherwise) has 
the same chance of being contacted. Obviously, this is 
more time-consuming than using the telephone book 
since non-working and non-household numbers are 
dialed too. The pollsters found that 84 per cent of the 
Ontario households surveyed had listed numbers. Ten 
per cent had unlisted numbers by request and the 
balance were not listed because of a recent change 
of address. 


Preliminary analysis showed that unlisted numbers were 
associated with young people, renters, large families, 
unmarried people, urban dwellers and lower standards of 
living. 


This finding is important for evaluators and their con- 
tractors who are designing telephone surveys. The poten- 
tial for bias in the results of directory-based sampling is 
worth worrying about. = 


OECD 
Public Management 
Activities 


The Organization for Economic Co-operation and 
Development (OECD) has an active program of work on 
public management issues which is relevant to evalua- 
tors. I have some general impressions from participating 
in the annual meeting of the Public Management 
Committee on April 8 and 9, and from discussions that 
followed it. 


Program Evaluation 


In addition to ongoing efforts, the Secretariat does spe- 
cial multi-year projects. For each project, the Secretariat 
develops an overall framework, identifies a number of 
case studies, and sometimes does a lessons-learned 
paper. It also provides more detailed advice to member 
countries. The main projects are: 


a) program/policy performance measurement; 
b) market type mechanisms (MTM); 
c) performance-related compensation; and 


d) plans for a program evaluation project. 


Program/policy performance measurement 


This work is led by the OECD’s Clifford Glover. The 
issues have been scoped and a survey has been sent to 
member countries. A number of European countries 
are becoming active and because of the very different 
approaches to delivery of programs, this project could 
provide some interesting and useful models and 
analyses. 


The principal limitation, from our perspective, might be 
that the European members seem a little too preoccupied 
by outputs and efficiency matters relative to questions of 
results, impacts and effectiveness. 


A second potential limitation is that the work might 
be too aggregated. There is, as I understand it, no plan 
to approach performance from the perspective of the dif- 
ferent programs and program instruments. I believe 
there is a need to be sensitive to the language and 
management of particular programs and programming 
instruments. 


The OECD Secretariat is aware of these potential limi- 
tations, but is responding to its more active clients. The 
OECD is quite interested in our continuing work in 
the area and we will be maintaining contact. We are 
pursuing work on conceptual and definitional issues, as 
well as more practical guidance and training in the defi- 
nition, measurements and reporting of program perfor- 
mance information. 


Market Type Mechanisms (MTM) 


The name MTM for this work might have a bit of an 
ideological bias toward private sector approaches to 
programming (“getting the job done”, “delivering the 
program”). The main focus of this work, however, is to 
expand thinking on alternative ways of achieving public 
objectives and finding out what has worked in what 
situations. 


The timing for such a project could not be be 
Resourcing pressures and PS2000 focus manag 
attention on results and services to the public, empi 
managers and staff to be creative in meeting dema 
and require all this to be done with decreased resou 
Departmental managers either have to improve the 
effectiveness of their activities or have some pretty : 
evidence that there is not a better way. 


This work is at an early phase, but some interes 
papers are being prepared. Francois Lacasse, whc 
a study for us on alternatives and cost-effectivenes 
evaluation, is leading this work for the OECD. He 


the link of the MTM work to effective program eva 


tion and is trying to organize the work around b: 
sectors and instruments of programming. 


The OECD’s version of the MTM concept includes \ 
ous forms of user fees, contracting out and subsi 
(grants to suppliers or vouchers to consumers). It 

includes organizational and resourcing alternati 
such as our special operating agencies (SOAs) and o 
variants. To illustrate this from one sectoral perspec 
(science & technology), it includes ideas such as 
creation of “intellectual property rights”, contracting 
private-sector funding and management of federal 
by providing combined intellectual property rights 

grants. 


Donald Lemaire of our office is keeping track of 
work. He will be trying to link these OECD efforts to 
own work in response to the need for more creative. 
practical alternatives of instruments and delivery 
evaluating programs. The government recently 
demonstrated an interest in such alternatives, suck 
the user fees and SOA initiatives. The MTM pro 
might also provide a framework for sharing lessc 
learned on which approaches to program delivery w 
best in what situations. In this way lessons learned fi 
evaluations in one situation might be made availabl 
managers and evaluators in other departments. 


Program Evaluation 


This work is likely to develop slowly, and is only in 

very early stages. As I understand it, there is no fi 
commitment to proceed. European experience is dive 
and more importantly there are great differences 
opinion among members of OECD about the meanings 
this term. 


The Secretariat staff were very cooperative and kno 
edgable. If you would like to find out about or contrib 
to this work, please let us know. Ronald C. Corbei 
actively involved in the performance work and can 
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reached at 957-7171. Donald Lemaire is heading the 

work on programming alternatives, which includes the 
MTM mechanisms, and can be reached at 957-7176. We 
look forward to hearing from you. = 


Martin Ulrich 


Mr. Ulrich is Director, Operations Division, Program 
‘Evaluation Branch, OCG. 
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Ein Capital Evaluators 
Capture CES Honours 


tits Annual Conference in May this year, the Canadian 
Evaluation Society honoured three members of the 
National Capital evaluation community for their contri- 
Don to the development of evaluation in Canada. 
Departing from past practice, the awards were spread 
over the three days of the conference. 


On the opening day of the conference, Guy Leclerc, 
leputy Comptroller General of Canada, Program 
ivaluation, received the CES Public Sector Award. This 

award recognizes an outstanding contribution to the 
establishment of evaluation in the public sector of 
vanada. Mr. Leclerc was singled out for his unstinting 
efforts over the last eight years to promote evaluation 
hroughout Canada, particularly for his dedication to 
establish program evaluation as a meaningful manage- 
ont practice at all levels of government. 


the society's highest award for professional excellence, 
he CES Award for Contribution to Evaluation in 
nada, was given the next day to Ronald C. Corbeil, 
senior Evaluation Analyst, Program Evaluation Branch, 
ce of the Comptroller General of Canada. This award 
recognizes an outstanding contribution to the practice of 
evaluation, the promotion and enforcement of standards, 
work as an evaluator and service to the society. In accept- 
ing the award, Mr. Corbeil acknowledged “. . . the support 
af the OCG, particularly that of Guy Leclerc, without 
Whom none of this would have been possible. Guy, half of 
his belongs to you!” 


Lim Aubrey, a doctoral student in clinical psychology at 
he University of Manitoba, was the happy winner of the 
Best Student Essay. His paper was on the evaluation of 
Jutpatient services to panic disorder patients at the 
dealth Services Centre in Winnipeg. By the way, Tim 
lails from Ottawa. 
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On the last day of the conference, John A. Burns, 
Partner and Practice Director for Program Evaluation 
and Comprehensive Audit, Ernst & Young was awarded 
the CES Award for Service to the Society. It honours 
those members who have made an exemplary contribu- 
tion to the service of the society. Among John’s many 
accomplishments are the conception and realization of 
the society's national student essay contest and the 
development of the practice of recognizing exemplary 
evaluation practices. 


The CES winners: John A. Burns, Guy Leclerc (seated), Ronald C. Corbeil 
(standing). 


Our congratulations to Leclerc, Corbeil and Burns. All 
three have laboured hard to build our community and 
the Society. = 


Peer Review and Evaluation 
at the National Research Council 


The National Research Council (NRC) is celebrating its 
75th anniversary this year. During those years, the 
Council has adapted its role to meet the needs of 
Canadians, changing from a primarily research-oriented 
organization to one with a research role coupled with an 
industrial role. The industrial role has been promoted 
more extensively over the last ten years. 


A 


Program Evaluation 


The Council’s mandate is “to undertake, assist, or 
promote scientific and industrial research”. As 
Canada’s leading government research agency, the NRC 
serves the needs of Canadians by helping to maintain and 
enhance the socio-economic well-being and competitive- 
ness of the country through: 1. the support of national sci- 
ence and engineering activities; 2. the performance of 
research and development; 3. the stimulation of invest- 
ment in research and developnment; and 4. the develop- 
ment and provision of vital expertise and knowledge. 


Traditionally, NRC assessed its activities using peer 
reviews. Peer review is a well-established practice for 
evaluating scientific work. Its key feature, as practised 
at NRC, is the assessment of the quality, relevance and 
performance of the Council’s scientific, engineering and 
technical endeavours, by external evaluators expert in 
the relevant subject areas. It encompasses the perfor- 
mance of both individual researchers and projects, and 
uses such factors as effectiveness, merit, scientific rele- 
vance, and future potential as assessment measures. 


The increased emphasis on an industrial role challenged 
the assessment process to extend beyond the peer review 
technique to examine the effectiveness of technology 
transfers made by NRC’s various research groups and 
measure their impact and effects. A formal program 
evaluation organization was established in 1981 to 
undertake a five-year-cycle evaluation process. At that 
time, peer review and program evaluation existed and 
reported separately and through different channels. 


In 1985, the assessment process consisting of peer review 
and program evaluation was integrated. At about the 
same time, an Assessment Committee of Council, com- 
posed of members of NRC’s Board of Directors (Council), 
was created with the responsibility for assessing NRC’s 
programs and reporting results to Council and the 
President. The Assessment Committee establishes 
Review Committees of Council to undertake the assess- 
ment of a program. These Review Committees consist of 
external experts in the field under review chaired by 
members of Council. The Program Evaluation group 
provides methodological expertise, collects and analyses 
information and assists in the preparation of reports in 
support of the review committee members. The use of 
such committees has revealed itself to be very effective 
in the transmission of evaluation results and recommen- 
dations to top management. 


NRC programs and activities are evaluated in a system- 
atic manner, consistent with the program evaluation 
policies and practices advocated by the Office of the 
Comptroller General of Canada. NRC evaluation 


employs a variety of analytical tools or methods. Tt 
methods range from surveys to determine the key 
tures of Canadian research capabilities in a given sec 
to selected case studies of client interactions to as 
the effectiveness of program delivery. In recent ee 
issues pertaining to the impact and effects of resea 
management practices have lead NRC to apply org: 
zational effectiveness and design methods. The over 
ing principle in the design and conduct of the evaluat 
is the need to be assured that the conclusions are b 
reliable and credible. Peer reviews are still underta 
parallel to other evaluation exercises. In the near fut: 
it is anticipated that peer review will be fully integra 
within the evaluation process as the primary tool 
evaluating issues related to the quality and relevanc 
the research. 


Further to the integration of the evaluation functic 
NRC has developed and located its program evaluat 
function so that its work can make an effective contri 
tion to the needs of corporate and executive mana 
ment. In the past two years, its role has been redefii 
and expanded to meet NRC’s strategic and operatio 
planning, policy development and corporate accounta 
ity reporting needs. These functions are now under: 
responsibility of a single director, thus allowing fo 
more comprehensive approach to program review £ 
providing an integrated set of management tools. 


Program evaluation is thus one element within the ov 

all assessment process at NRC. Combined with p 
review and audit, program evaluation provides a ba 
for a comprehensive assessment of NRC’s research act 
ities in a manner that is both appropriate to a creat 
research environment, and relevant to the managems 
needs of the organization. s 


Robert Letell 


Mr. Letellier is Head, Program Evaluation at 1 
National Research Council. 
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Evaluation is Spreading 


The Inland Revenue Department of New Zealand will be 
establishing its own program evaluation function later 
this year. The start-up of an evaluation group, the first 
in that country, has been facilitated by the efforts of 
William Baker, Acting Director General, Corporate 
Affairs Branch, Revenue Canada, Taxation and past 
president of the Ottawa Chapter of the Canadian 
Evaluation Society. William took the opportunity in a 
recent visit to New Zealand to promote the merits of pro- 
gram evaluation and developed guidelines for its intro- 
duction. 


Revenue Canada, Taxation has been requested to assist 
in the implementation and will be sending an experi- 
enced evaluator to participate in Inland Revenue’s first 
evaluation study. Daryl Young, a senior analyst who 
has been in the evaluation community for eight years, 
will be leaving for Wellington in late August and is 
expected to be on assignment for five months. m= 
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Practicum, March 22, 1991 


“Service to the Public, Measuring Performance - 
How can this be achieved?” 


Restraint in the 90s, PS2000 and accountability are 
three of the main reasons why measuring and monitor- 
ing service to the public continues to increase in impor- 
tance in the federal Public Service. 


This was the dominant theme running throughout the 
“Practicum” session held this spring by the Program 
Evaluation Branch of the Office of the Comptroller 
General. Senior managers in finance, audit and evalua- 
tion as well as line managers attended the half-day ses- 
sion, designed to focus on developing useful performance 
indicators in an age of “bottom line” results management 
and ever-decreasing resources. The panel discussion was 
based on an OCG draft paper entitled “Measuring and 
Monitoring Program Performance and Service to the 
Public”, a document which focuses on the disciplined 
development of indicators of performance and measure- 
ment strategies, designed to convey in a succinct and 
credible manner how well a program is performing. 


Chaired by Guy Leclerc, Deputy Comptroller General, 
Program Evaluation Branch, panel members also 
included G. Neil Faulkner, Assistant Deputy Minister, 
Medical Services Branch, Health and Welfare Canada; 
Ranald Quail, Associate Deputy Minister, Transport 
Canada; Don Rennie, Coordinator, Communication 
Management Practices, Privy Council Office; and Dr. 
John Mayne, Director, Evaluation Policy (OCG). 


The private sector experience was provided by Bob 
Weese, currently on a Business Government Executive 
Exchange from Supply and Services Canada as Manager, 
Corporate Business Development Program, at GE 
Canada. Weese’s presentation centred around the GE 
corporate premise that speed, simplicity and self-confi- 
dence are the keys to delivering service to the public. 
Said Weese, “Customer service measurement starts with 
a definition of what to measure. Then performance stan- 
dards must be established, followed by ongoing internal 
and external processes for measuring performance. The 
key is that measurements must be simple and credible.” 


Weese outlined how GE medical systems measure cus- 
tomer satisfaction through annual surveys of client hos- 
pitals, and builds the results into the compensation 
package for all employees of the business. 
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Weese also supported the need to develop internal indi- 
cators to support external performance measures, while 
at the same time recognizing that some key elements of 
client satisfaction, such as image, quality and respon- 
siveness, are difficult to measure quantitively on an 
ongoing basis. 


In his remarks to the group, John Mayne stressed the 
importance of adopting an orderly approach to devel- 
oping performance indicators. “It is important not to try 
to measure too much,” he stated. “A balance is needed 
between indicators which measure delivery outputs and 
those of a more strategic nature. A comprehensive pic- 
ture of program performance covers the relevance, 
success and cost-effectiveness of the program.” 


Dr. Mayne pointed out the need to distinguish between 
performance indicators and performance expectations, 
with the latter linked to the program objectives. Both 
numerical targets and qualitative descriptions can be 
used to define expectations as well as long- and short- 
term expectations. 


“Program performance expectations stated in terms of 
results, rather than activities, provide flexibility in 
choosing how to deliver the program. To find out what 
clients expect and to feedback on what an evaluator can 
accomplish requires client consultation. Consultation 
serves to better understand respective positions,” Mayne 
added. He also noted that service level targets must not 
only be challenging enough to motivate staff, they must 
also be achievable. 


Mayne outlined a number of lessons learned in develop- 
ing service level indicators. This information is now 
available in the OCG publication Line Managers and 
Assessing Service to the Public. 


He concluded by emphasizing that performance infor- 
mation must be for managers, not information systems; 
that evaluators, auditors or other performance measure- 
ment professionals should serve as facilitators and that 
it should be kept in mind that the clients of programs 
have an important perspective on program performance. 


In his remarks about measuring service to the public, 
Ranald Quail stressed the “service agency” aspect of 
Transport Canada. He noted that while PS2000, with its 
emphasis on service to the public, involves the pushing 
down of authority, or empowerment, it also by definition 
involves risk-taking and innovation. “But these notions 
also mean trust, taking off the strictures and relying on 
the service relationship to tell you whether the trust is 
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justified — a don’t-shoot-the-messenger approach w 
our people and an open, frequent and well-underst 
relationship with clients.” 


Quail added that the consultative process, especially 
policy, will be delicate. However, it should be routine & 
frequent. He noted that consultation does not necess 
ily mean consensus, but it must be perceived as be: 
meaningful, genuine and capable of making a differen 
Within a service department, it is the people at the del 
ery end of the process who must make it all happen, “a 
we have instant feedback from the travelling pul 
everyday as to the job we’re doing.” 


In addressing the issue of service to the pub 
Don Rennie of the Privy Council Office stated that p 
haps public service managers were the victims of 1 
much data and not enough useful information to be 
effective as they might be in delivering service to t 
public. He noted it is not just management which 
involved in service delivery. There are 218,000 ott 
Public Service employees on the front lines. They 1 
must be involved if the right messages are to be del 
ered. A common mission or sense of values is essential 
direction and commitment and to make sure taxpaye 
get good value for their money. Public Service renev 
means encouraging innovation and calculating ris 
Rennie noted, and it means rewarding innovation a 
creativity in some tangible way in order to encoura 
managers to engage in a little risk-taking for the bene 
of better service to the public. 


Health and Welfare Canada’s perspective on service 
the public was put forward by Neil Faulkner, who stat 
that it is important to implement regionally-flavour 
programs, but that by definition these are difficult to r 
up into a collective picture. He noted that another dif 
culty in improving service delivery is data that, becau 
it’s inconsistent or incomplete, cannot be accepted by : 
the stakeholders. “Evaluation implies a review of tl 
past,” stated Faulkner. “In order to be meaningful, evs 
uation needs to be ongoing and performance indicato 
need to reflect current trends, so that they can assist 
the development and design of new programs 
International comparisons can be helpful in some cas 
as well, be added. = 


Nancy Marley-Clark 


Mrs. Marley-Clarke is Communications Analyst with tl 
Program Evaluation Branch, OCG. 
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CES Annual 
vonference 1991 


most 400 participants from across the country 
ittended the Canadian Evaluation Society’s (CES) 12th 
nnual conference in Vancouver from May 5 to 8, 1991. 
fhe theme of this conference was “The Politics of 
ivaluation: A Challenge to Ethics, Issues and Skills.” It 
vas both well-organized and well-socialized. The wel- 
oming reception on the evening before the conference 
fficially started and the coffee and muffins every morn- 
ng were especially appreciated. The receptions hosted 
ly the chapters presented participants with a welcome 
portunity to network more broadly, while the wine 
Valuation hosted by the British Columbia Wine 
nstitute gave all of us a chance to “evaluate” how well 
ur palates compared to those of our peers. 


. 
he 


Beg 

na departure from tradition, full- and half-day work- 
hops were held throughout the three days of the confer- 
nee for those who wanted a deeper understanding of 
er in topics. And there was a full range of topics, from 
eneral to technical. The former included neutralizing 
dlitical pressures, using evaluation for change, plan- 
ing evaluations, clearer writing, internal evaluation, 
erformance measurement, questionable evaluation 
ractices, information technology evaluation and evalu- 
ting not-for-profit programs. The more technical topics 
Wered questionnaire design, logic models, measure- 
lent, sample sizes and publishing evaluation findings. 
hey even provided a host of sessions on analysis: multi- 
ariate, factor, multi-attribute, data, and the perennial 
st-benefit. Holding workshops over the three-day 
riod proved to be a huge success as most of them were 
er-subscribed. 


nother departure from tradition was the presentation 
‘the society's awards. They, too, were spread over the 
ree days of the conference. As most of them went to fed- 
al evaluators, a separate article on page 7 has been 
voted to them. 


ne first plenary session, on Monday morning, featured 
r. Ernest House, a professor in the School of 
ducation at the University of Colorado at Boulder. He 
Iked about the influence of muliticulturalism on eval- 
ition, illustrating his points, in part, with current 
velopments in Quebec. Actually, Dr. House’s contribu- 
m to the conference started a day and a half earlier. 
ouse, the former editor of New Directions for Program 
aluation and the 1990 recipient of the American 
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Evaluation Association’s Lazersfeld Award for 
Evaluation Theory, conducted a very successful pre- 
conference workshop on “Problems in Evaluation: 
Multicultural and Social Justice Issues” starting 
Saturday afternoon and finishing Sunday evening. 


At lunch, Dr. Ken Rubin, president and co-founder of 
Apogee Research, Inc., talked about his evaluation work 
with the Environmental Protection Agency in the United 
States. Evaluation in this field is growing at a phenome- 
nal rate and covers matters such as water supply sys- 
tems, waste water treatment plants, solid and hazardous 
waste management facilities and water resource devel- 
opment projects. 


At the plenary session the next day, Dr. Alex Michalos, 
editor-in-chief of The Journal of Business Ethics and pro- 
fessor of philosophy and social sciences at the University 
of Guelph, took a slightly tongue-in-cheek look at the 
society's claims to “ethical research” and “independence 
and objectivity.” Adopting an ontological approach, this 
winner of the Secretary of State’s Award for Excellence 
in Interdisciplinary Studies in the field of Canadian 
Studies laid bare the frailty of these concepts and 
advised caution in their use. 


Immediately following the plenary session, the council 
executive, its various committee chairs and chapter rep- 
resentatives reported on the state of the society at the 
Annual General Meeting. Briefly, membership grew by 
20 per cent over the year, to stand at about 1,350; all the 
chapters are active and growing; and the society, due to 
the growth in its financial reserves over the last few 
years, was able to weather a substantial dip in its equity. 
Alan Winberg, director, Administrative Management 
Policy and Evaluation, Treasury Board Secretariat, is 
the new president. 


The final plenary speaker of the conference was Senator 
Lorna Marsden. In a thoughtful, carefully constructed 
luncheon address on the closing day of the conference, 
she articulated and illustrated three precise situations 
in which program evaluation could help politicians. 
First, at the introduction of legislation for either pro- 
grams or organizations, second, when program funding 
is cut or substantially increased, and finally, at a time of 
crisis. Her conclusion, however, was that “...program 
evaluation is of great help to politicians but as it is cur- 
rently organized and used, more in the promise than in 
the delivery.” Copies of the Senator’s address (as well as 
those of Drs. House and Michalos) can be obtained 
by writing to Kathy Jones, Executive Secretary, 
309 James Street, Ottawa, Ontario, K1R 5M8 or calling 
her at (613) 230-1007. 


le RE es 3 sty Me ai a ta 
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As in previous years, federal evaluators and officials 
made a strong contribution to the conference. Contri- 
buting to various workshops, panels and other sessions 
were: Carol Motuz (Agr); Gilles Blanchet, 
Francoise Mailhot (CIDA); Henno Moenting (CSIS); 
Shelley Borys, Erica Claus (Com); Nand K. Tandan 
(EIC); Ibolya Galambos, Ike S. Jaswal, Langley Muir 
(EMR); Paul Finn, Edward Overstreet, Claude 
Théorét (HWC); Daniel J. Caron, Beverley Clarkson 
(INAC); Michael O’Brecht (MRC); George Teather 
(NRC); Neil Maxwell (OAG); Ronald C. Corbeil, 
Guy Leclerc, John Mayne, Toru Uno (OCG); 
Avrim Lazar (PSC); Peter Wiebe (Sec); 
Michel Bouffard, Mireille D’Auray-Spattz, Alan 
Winberg (TBS); Ken Chatfield, Jon Fisher (TC); 
Mike Charles (VAC). 


Ronald C. Corbeil 


Mr. Corbeil is Senior Evaluation Analyst of the 
Evaluation Operations Division (OCG) 


The Report on 

Federal Program Evaluation 

of the Senate Standing Committee 
on National Finance 


Between March and November 1990, the Senate 
Standing Committee on National Finance held 10 hear- 
ings on federal evaluation, originally motivated by the 
views of Prof. D.G. Hartle of the University of Toronto. 
Hartle had argued that the Senate should, on a selective 
basis: (i) put evaluative questions to ministers or (ii) com- 

. mission evaluations (or interpretations of existing eval- 
uations) from persons or groups outside the government 
with conflicting interests or values. These suggestions 
were intended to increase accountability in government. 
The committee rejected these proposals early on in its 
deliberations but sought ways to make departmental 
evaluations more useful to it, particularly in its annual 
review of the Estimates. 


The committee was provided with extensive documenta- 
tion on the utilization of evaluation by all federal depart- 
ments and agencies. (This material is being compiled 
by the Office of the Comptroller General and will be 


published in a Compendium of Evaluation Utilizati 
this summer.) In addition to hearings with senior m: 
agers and evaluators from four federal departments, 

OCG, the Office of the Auditor General, and with Pi 
Hartle, the committee heard from the United Sta 
General Accounting Office and from Canadian eval 
tors in the private sector and in the Government 
Alberta (the current President of the Canadi 
Evaluation Society). The Newsletter of the Canadi 
Evaluation Society has carried excerpts of the testimc 
in recent issues. 


The committee’s main conclusion was that the presi 
system of evaluation should be maintained but should 
strengthened in order to improve the accountability 
the government to Parliament. In its view, there is ins 
ficient objectivity and independence from line mana 
ment in current evaluation practice. In that context 
recommended that present departmental evaluators 
made employees of the OCG, with a status similar 
departmental lawyers who are employees of 
Department of Justice. This recommendation was a 
intended to counteract the perceived excessive dec 
tralization of federal evaluation as a result of the IM 
and PS2000 initiatives and to give the OCG dir 
responsibility for conducting program evaluations. 


The OCG has welcomed the committee’s report, 1 
attention it has given to federal evaluation in hearir 
and otherwise over many months and its grappling w 
the difficult issues of how to make federal evaluati 
more effective. The OCG, however, has not been able 
support the main recommendation of the committee. 
was felt that the recommendation would have the eff 
of centralizing evaluation resources as a vehicle 
accountability of departments to central agencies a 
Parliament. This was somewhat at odds with 1 
intended aim of federal evaluation as a departmen 
senior management tool and the increasing responsib 
ties being given to departments and managers as a res 
of the IMAA and PS2000 initiatives. 


Cameron McQue 
Mr. McQueen is Acting Director of the Progr 


Evaluation Division, Revenue Canada — Custo 
and Excise. 
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Agriculture Canada 
Carol Motuz 994-1991 
Director, Program Evaluation Division FAX 994-1782 


Atlantic Canada Opportunities 


ency 
aaa Tripp (506) 851-6515 


Director General, Audit FAX (506) 851-7403 
and Evaluation 


Atomic Energy Control Board 
James Elks 995-6535 
A/Director, Planning and Coordination FAX 995-5086 


Canada Council 
Peter Brown 598-4311 
Senior Assistant Director FAX 598-4390 


and Treasurer 


Canada Mortgage and Housing 


Corporation 
Craig Angus 748-4659 
AlDirector, Program Evaluation FAX 748-6192 
Division 


Canadian Centre for Occupational 


Health and Safety 
Kash Manchuk (416) 572-4401 
Director, Finance, Publishing FAX (416) 572-2206 


and Administration Services 


Canadian Broadcasting Corporation 

Ron Sweeney 738-6592 

Director, Coordination of Planning FAX 738-6666 
and Evaluation 


Canadian Centre for Management 


Development 
Louise Durocher 992-8061 
Director, Corporate Services FAX 995-0331 


Canadian Commercial Corporation 


F. Owen Kelly 996-0116 
Comptroller FAX 995-2121 
Canadian Dairy Commission 
Nelson Coyle 998-9490 
Policy and Program Analyst FAX 998-4492 
Canadian Human Rights 

Commission 
Harry Goldberg 943-9090 
Director, Research and Policy Branch FAX 996-9661 
Canadian International 

Development Agency 
Jean Quesnel 997-0148 


Director, Program Evaluatiän Division” FAX 953-6356 


Canadian Radio-television and 
Telecommunications Commission 

Denis O'Rourke 997-9414 

A/Director, FAX 994-0218 

Corporate Audit and Evaluation 


Canadian Security Intelligence 


Service 
Joanne Weeks 954-1464 
Special Studies Branch FAX 952-7565 


Communications Security 


Establishment 
Donald Drew 991-7315 
Director, Evaluation and FAX 991-7323 
Audit Services 


Commissioner of Official Languages 


Yves Breton 995-0815 
Advisor FAX 993-5082 
Consumer and Corporate 

Affairs Canada 
Michael Wernick 953-6735 
Director, Strategic Planning and FAX 953-6660 


Corporate Services 


Correctional Service Canada 
Doug Borrowman 995-9000 
Director, Evaluation and Research FAX 995-3603 


Department of Communications 
Francine Chabot-Plante 990-4735 
Director, Program Evaluation Branch FAX 952-1208 


Department of Finance 
David Sewell 992-4309 
Director, Tax Measures FAX 992-2571 


Evaluation Division 


Emergency Preparedness Canada 

Michael Martin 991-7703 

Director, Corporate Planning FAX 996-0995 
Coordination 


Employment and Immigration Canada 

Tan Midgley 994-2406 

Director General, Program FAX 953-8584 
Evaluation Branch 


Energy, Mines and Resources Canada 


Marcel Gibeault 996-4940 

Director General, Audit and FAX 995-5719 
Evaluation Branch 

Environment Canada 

Leif Skulstad 997-6946 


A/Director, Program Evaluation FAX 953-2107 


Departmental Contacts for 
Program Evaluation 


Export Development Corporation 


Terry Burbridge 598-2870 

Manager, Operational Planning FAX 237-2690 
and Development 

External Affairs Canada 

Bernard M. White 990-9840 

A/Inspector General, Audit and FAX 952-3904 
Evaluation 

Farm Credit Corporation 2 

Michel Messier 996-6606 

Director, Program Evaluation FAX 996-8533 


Federal Business Development 


Bank 
Jacques Lagacé (514) 283-7300 
Vice-President, Internal Audit (514) 283-0617 
and Program Evaluation 
Fisheries and Oceans 
Robert Bergeron 993-1937 


Director, Program Evaluation Branch FAX 996-9055 


Forestry Canada 

Bob Lahey 997-1107 

Director, Audit and Evaluation FAX 995-1037 
Directorate 

Health and Welfare Canada 

Beverley Holloway 957-3887 

Director, Program Evaluation FAX 957-8910 

Indian and Northern Affairs Canada 

Marie-France D’Auray-Boult 994-4311 

Director, Evaluation Directorate FAX 994-1283 


Industry, Science and Technology 


Canada 
Byd McBain 954-9633 
Director, Evaluation Directorate FAX 954-1894 
Investment Canada 
C. Edward Hitsman 995-3452 
Director, Corporate Services FAX 996-2515 


International Development 


Research Centre 
Douglas Daniels 598-0593 
Director, Office of Planning FAX 238-7230 


and Evaluation 


Justice Canada 

Michel Vallée 993-5003 
Director, Program Evaluation FAX 992-6340 
Labour Canada 

Denise Roberge 994-0728 
Director, Corporate Services FAX 997-1664 
Medical Research Council 

Michael 0’Brecht 954-1817 
Program Evaluation FAX 954-1800 
National Capital Commission 

Carman Young 239-5082 
Director, Management Audit FAX 239-5695 


and Evaluation 


National Defence 
Paula Skippon 996-4717 
Director General, Program Evaluation FAX 992-0528 


National Energy Board 
Claire Scott 990-6845 
Director, Planning and Review Branch FAX 990-7900 


National Film Board 

Claude Parent (514) 283-9251 

Director, Planning and Program FAX (514) 496-1646 
Evaluation 


National Library 
Jean Higginson 995-5026 
Senior Policy Analyst, Policy FAX 996-7941 


and Planning 


National Parole Board 
Pierre Couturier 954-6127 
Director, Evaluation and Statistics FAX 995-4380 
National Research Council 
Robert Letellier 990-9551 
Head, Program Evaluation FAX 952-9907 
National Search and Rescue 

Program 
David Walters 996-1696 
Director, Audit and Evaluation FAX 996-3746 


National Transportation Agency 


Gilles Tardif 997-1260 
Director, Financial Planning and FAX 953-8623 
Administration Directorate 


Natural Sciences and Engineering 


Research Council 
Jocelyn Lillycrop 995-5911 
Chief, Evaluation and Audit FAX 946-0742 
Office of the Auditor General 
Ted F. McNamara 995-4601 
Assistant Auditor General, FAX 957-4023 
Program Evaluation and 

Internal Audit 


Office of the Superintendent 

of Financial Institutions Canada 
Andy Poprawa (416) 990-7797 
Director, Program Review FAX (416) 973-7021 


Public Archives Canada 

Ray Maddocks 996-7245 

Director, Program Evaluation FAX 992-9010 
and Research 


Public Service Commission 


of Canada 
Suzanne Peters 992-8564 
Director, Program Evaluation FAX 995-6044 
Public Service Staff Relations Board 
Louis Labrecque 990-1825 
Director of Administration FAX 990-1849 
Public Works Canada 
Barry Bragg 736-2080 
Director, Program Evaluation FAX 736-2099 
Revenue Canada, Customs 
and Excise 
Cameron McQueen 954-7881 
A/Director, Program Evaluation FAX 952-6770 
Division 


Revenue Canada, Taxation 

asil Orsini 954-6042 
A/Director, Internal Audit and Program FAX 952-7476 
Program Evaluation 


Royal Canadian Mounted Police 


Donald Coates 993-3633 
Senior Evaluation Officer, Program FAX 993-4453 
Evaluation Branch 
Secretary of State 
Kerry Johnston 994-1224 
Director, Program Evaluation FAX 953-2792 
Social Sciences and Humanities 
Research Council 
Rebert Hanson 992-5148 
Director, Evaluation and Statistics FAX 992-1787 
Division 


Solicitor General of Canada 


Clay Ellis 991-2918 
Director, Management Review FAX 990-9077 
Statistics Canada 

Martin Hubbard 951-8291 


Director, Program Evaluation Division FAX 952-9099 


Supply and Services Canada 


Allan Macmillan 956-2975 
Director, Evaluation Branch FAX 956-5868 
Supreme Court of Canada 

Irene M. O'Connor 996-7662 
Chief of Finance FAX 996-7266 
Telefilm Canada 

Robert Armstrong (514) 283-6363 


Director, Research and Analysis FAX (514) 283-8212 


Transport Canada 

Jon Fisher 993-4418 

A/Director General, Program FAX 954-1993 
Evaluation ; 


Treasury Board of Canada 
Office of the Secretary of 
the Treasury Board 
Doug Redden 957-2419 
A/Director, Corporate Policy FAX 952-3658 
and Planning 


Personnel Policy Branch 

André Robert 952-3141 

Director, Evaluation and Review FAX 954-1018 
Group 


Administrative Management 


Branch 
Alan Winberg 957-0743 
Director, Administrative FAX 952-1381 


Management Policy Evaluation 


Official Languages Branch 
Gérard Finn 952-2864 
Director, Policy Division FAX 952-2862 
Treasury Board of Canada 
Comptroller General 
Evelyn Levine 957-7071 
Director, Corporate Management FAX 952-8536 
Services 
Veterans Affairs Canada 
Dick St. John (902) 566-8017 
Director, Program Evaluation FAX (902) 566-8051 
Directorate 


Western Economic Diversification 
Jim Swan 952-0676 


Director, Internal Audit and Program FAX 952-9384 
Evaluation 


GI 
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IIGHLIGHTS: 


leads of Evaluation 
leeting: Chaffey’s Locks 1 


On June 27-28, 1991, some 30 heads of 
evaluation and 15 staff members of the 
Office of the Comptroller General 
(OCG) met to discuss how depart- 


he Treasury Board of Canada 
mental evaluators and the OCG could 


dopts a New Program 


valuation Policy 9 combine their efforts to respond more 
: effectively to the challenges facing the 
ier Il Audit -OAG program evaluation function. The 


meetings were co-chaired by Alain 


he Need for Better Barbarie, Director of Evaluation (until 
erformance Measurement 7 — July of this year), Energy, Mines, and 
xploring Alternative Rogan and Ronald C. Corbeil of 
odes of Intervention: 

earning from Evaluation 8 The aim was to develop a “shared 


3 community agenda”, for community 
action on important issues. Although 
some follow-up has already occurred 
kK and is described elsewhere in this 
à newsletter, we plan to publish a special 
; edition summarizing the discussion and 
some of the actions identified for 
horter n follow-up. 
ous or Zo np shorter and longer term follow-up 
The meeting opened with an address by 
J. Andrew Macdonald, Comptroller 
General of Canada. He outlined his 
views on the challenges facing the 
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program evaluation function, as well 

as the attendant opportunities. In 
particular, he focused on the implica- 
tions of Public Service 2000 and the 
continuing resourcing pressures as well 
as the impact they will likely have on 
expectations for program evaluation. 
Detailed notes on his comments will 

be available in the special issue. 

Maria Barrados and Jim Blain of the 
Office of the Auditor General completed 
the opening session with a description 
of their plans for the Tier II audit of 
effectiveness (including program 
evaluation) to be reported in 1993. A 
fuller description of their presentation 
is described later in this newsletter. 


Most of the time during the meeting was 
allocated to small-group discussions 

of issues identified by a committee 

of heads of evaluation and by the 
Government Program Evaluation 
Perspectives: Into the 90's. The agenda 
covered: 


¢ Reporting on Performance 

e Evaluation and Internal Audit 

¢ Development of Evaluators 

e Interdepartmental Evaluations 

e Alternative Programming 
Mechanisms 


Program Evaluation 


Service to the Public 

Evaluation Thrusts and Priorities 
Performance Indicators 

The Role of the OCG 


The meetings concluded with plenary reporting on the small- 
group discussions. Doug Wood, Acting Deputy Comptroller 
General, Evaluation and Audit Branch, OCG, closed the 
meeting with a commitment to act on those aspects within 
OCG control and to continue to work with the evaluation 
community on the shared community agenda. = 


Ronald C. Corbeil 


Mr. Corbeil is a senior evaluation analyst with the 
Evaluation and Audit Branch, OCG. 


TheTreasury Board of Canada 
Adopts a New Program 
Evaluation Policy 


In August, the Treasury Board adopted a new program 
evaluation policy. This policy represents the culmination 
of extensive consultation with federal departments and 
agencies, and is intended to ensure that the evaluation 
function becomes an essential management tool to the 
federal government in the 1990s. 


The previous evaluation policy (Treasury Board Circular 
No. 1977-47) had been in force for more than a decade and 
needed revision and update. Since the mid-1980s, various 
initiatives, such as Increased Ministerial Authority 
Accountability, PS2000 and the Shared Management Agenda 
have been introduced to make the federal administration 
more accountable, more efficient and more responsive to 
the public. These projects, together with severe fiscal 
restraints, have significantly altered the management 
environment. The current thrust is toward a decentralization 
of responsibilities, increased focus on results and the 
introduction of a new kind of relationship between central 
agencies and line departments — one that is less based on 
control and more on cooperation. It was important that the 
program evaluation policy reflect this new way of doing 
business. 


It was equally important that the new version of the policy 
build on past experience and not change proven formulas. 
Consequently, the basic policy remains the same: deputy 
heads are still responsible for evaluating their programs and 
for confirming or modifying these programs in the light of 
evaluation findings. 


The difference is that the new policy gives departments and 
deputy heads more latitude, especially in terms of organizing 
their evaluation function. This should enable them to better 


adapt evaluation practices to their specific needs. There is 
no longer a requirement for wall-to-wall coverage of the | 
organization’s activities. In the future, departments will b. 
responsible for evaluating their programs but will need to! 
conduct such studies only when they will be material and! 
cost-effective. The 1991 policy also states more emphatic¢ 
that evaluation is part of the strategic management of | 
federal institutions. The respective responsibilities of the | 
deputy heads, the Treasury Board Secretariat (TBS) and t3 


~ OCG with respect to program evaluation have been defin d 


more clearly and precisely, thereby promoting greater 


accountability. 


The key points of the new Treasury Board policy on 
program evaluation are: 


Policy Objective: 
To ensure that federal departments have relevant, credibl 
and objective information available on the performance 07 
their programs and that they use that information for the | 
cost-effective and accountable management of programs. 


Policy Statement: 


It is government policy to periodically evaluate the 
continued relevance, success and cost-effectiveness (i.e., ! 
program performance) of federal programs and use that | 
information to reconfirm, improve or discontinue progran 


Policy Requirements: 


(a) Responsibilities of the deputy head: the deputy headi i 
responsible for: | 
e reconsidering programs based on evaluation finding 
and using such information to reconfirm program _ 
activities, improve their cost-effectiveness, | 
discontinue them or recommend appropriate progra 
changes to the minister or Cabinet; | 
° reporting on program performance to the OCG, the | 
TBS, Cabinet committees and Parliament, as agreed 
or requested; | 
e directing evaluation by making resourcing provision 
for evaluations; | 
e establishing evaluation criteria by ensuring that | 
performance indicators and expectations are set for 
all programs; | 
e strategic planning of evaluations to meet department 
priorities and the information needs of various Cabii 
committees; and 
e evaluating departmental programs periodically whe 
such evaluations are material and cost-effective. 


(b) Joint responsibilities of the TBS and the OCG: they ai 
jointly responsible for: | 
°_communicating Cabinet concerns and priorities with 
respect to evaluation; 

e review of evaluation criteria contained in Treasury 
Board submissions; 

e review of program performance reports; and 


¢ overseeing central evaluations requested by Treasury 
Board or any other Cabinet committee. 


(c) Specific responsibilities of the OCG: the OCG is 
responsible for providing leadership and support 
in program evaluation throughout the federal 
government by: 
° providing assistance and advice; 
¢ assessing evaluation practices; 
¢ fostering a professional evaluation community; and 
¢ coordinating the planning of departmental evaluations. 


Monitoring: 

The OCG will monitor the performance of rene in 
- meeting the requirements of this policy using readily 
_ available information and reports. Monitoring will assess 
_ both the quality of departmental evaluation and the quality 
_ of key evaluations under the OCG Working Standards for 
_ the Evaluation of Programs in Federal Departments and 
_ Agencies (OCG, 1989). 


Copies of the new program evaluation policy and related 

. guidelines can be obtained, in both official languages, from 
the Evaluation and Audit Branch, Office of the Comptroller 
_ General, by calling (613) 957-1928. m 


Marc Lemay 


“Mr. Lemay is a policy analyst with the Evaluation and 
Audit Branch, OCG. 


Welcome’ Session _ 


. The lon and Audit Branch of the Office of ne 


_Conpirler General (OCG) held its 3rd “Welcome” session _ 


for program evaluators on November 22, 1991. The … 
urpose of these sessions is to give new evaluators an __ 
Opportunity to learn about the role of the OCG in regard to _ 
_ the evaluation function and to meet with AUS of the 
Fe Evaluation and Audit Branch. 


E Doug Wood, Acting Deputy Comptroller General 
Evaluation and Audit Branch, led the session. 
| Mr. J. Andrew Macdonald, Comptroller General of 
« Canada, made some welcoming remarks. Cameron 
. McQueen, Senior Evaluation Policy Analyst; Al Whitla, 
Director, Audit Policy Division; Martin Ulrich, Director, 
_ Operations Division; Libby MacRae, Acting Director, 
- Community Development and Communications Division; 
_ and Garry Sears, President of the National Capital 
. Chapter of the Canadian Evaluation Society, were the 
a “hosts”. 


Paulette Somerville 
Editor 


: Asking the Right - 


Program Evaluation 


describe the a of: surveys. Even the. news ncaa 
- frequently report that surveys are to be considered 

_ accurate within some percentage. But this may be 

. providing a false sense of security to evaluators and 

… those who hav to use: their findings. — ee 


- The eo. 1991 Newsletter of the York University 
Institute for Social Research describes a fascinating 
_ experiment. Half (chosen randomly) of a surveyed 
| group were asked if they favoured stores being allowed 

_ toopenon Sunday. The other half were asked if they 

_ favoured a “common pause” ’ day (where most stores 
_ would be required to close). These questions are 
. logically almost equivalent in that one cannot 
. consistently favour both. Yet 73 per cent of the first 

_ group wanted open Sundays and 44 per cent of the - 

_ second (presumably matching) group wanted closed 
. Sundays. It seems that the framing of the question 
_ affected the responses or even created the opinions. In 

fact, ina follow-up question, fully a third indicated that 


they didn't really care! However, they were still willing 


to volunteer : an opinion. 


Think of the Dion i: a survey on the quality of a 


government service program. Will asking “Are you 


_ Satisfied?” get the same answer as asking if the client 


had any problems? Maybe you should ask both. Such a 


_ parallel survey could give more insight into the 
peparent precision of the findings. 


Another threat to the validity of SINKS was illustrated 
vividly during the controversy about the nomination _ 
of U.S. Supreme Court Justice Clarence Thomas. Polls 


reported on TV showed that the proportion of 
- women prepared to tell pollsters they had been 


sexually harassed almost tripled in the week of the 
U.S. Senate hearings. Obviously, high-profile events 
can have dramatic and sudden impacts on public 
perceptions. This argues for spreading surveys over 


_ time, looking for trends and discontinuities or doing 


small confirmatory surveys early and late in the 
evaluation. @ 


John Batchelor 


Dr. Batchelor is a senior evaluation analyst 
with the Evaluation and Audit Branch, OCG. 


Program Evaluation 


The Measurement 
of Program 
Effectiveness and 
the Reporting and 
Use of Effectiveness 
Information 


Introduction: 


A report on a government-wide audit of effectiveness 
measurement is planned for 1993. The audit will examine 
the adequacy of effectiveness measurement, reporting and 
use with respect to federal programs. Included will be: 


1) A series of case studies on individual programs selected 
to reflect the diversity of program types and the 
departments that administer them. The case study 
analysis will focus on examining the adequacy of the 
range and quality of effectiveness measurement (both 
ongoing and periodic) for each program in question, and 
the reporting and use of this information. The case study 
work will be coordinated with entity audit teams already 
conducting scheduled audits and then will be reviewed in 
aggregate. In cases where effectiveness measurement, 
reporting or use is found to be inadequate, the 
implications of the deficiency will be examined. 


2) A government-wide analysis of program evaluation 
coverage related to program activities as identified in 
Part III of the Main Estimates. 


3) An audit of the responsibilities of the OCG for program 
evaluation (based on the findings from the steps 
described above — in the context of the OCG’s 
responsibilities for the operational implementation of 
government policy on program evaluations. 


Audit Objectives: 


To determine the extent to which program effectiveness is 
satisfactorily measured and reported and the information 
utilized, where reasonable and appropriate. 


To determine the significance of any failure to satisfactorily 
measure, report and/or utilize program effectiveness 
information, where such measurement is reasonable and 
appropriate. 


Audit Scope: 


This is a government-wide audit with the scope focused 
upon program outcomes. The assessment of the adequacy of 
the measurement, reporting and use by departments of 
information on program effectiveness will include both 
periodic and ongoing measurement activities. 


2 The 6th Annual Worahon at the National Capital Che 
: {Canadian Evaluation Society (CES) was held on Noy 


| aos was “Three Current Issues: A Fos foe 


The on Hotel a en change Je 


te ‘The new formats is intended to enhance hele 
_ the ee by rs them a ins pa 


e detailed analysis of the reaction to then new fo 


D: in cena of tive at forma was | 
keynote and luncheon speakers. This year, Mr. Otto _ 
 Brodtrick, Principal, Office of the Auditor General, lai 
: pci address and Professor Alan pee Pi 


Proposed Work Schedule: 
— Individual case studies 1991, 1992, 19€ : 
— Government-wide evaluation coverage 1992 
— Analysis of case studies 1993 
— OCG audit 1993 
Maria Barrados/Jim Bla 


This article was submitted to us by Ms. Barrados and 
Mr. Blain, from the Office of the Auditor General of 
Canada. 


: + omis . The event over 2 Bei Tr 


Zi ee 


; Instead of the conventional Date by y : 


Lie of format. The organizers will be Bet sunt 4 


ie Frizzell s luncheon speed covered t tl he co He 


be à politi Cue 


D. Uno is a senior coakiataone analyst of no perati 
hort Dace: and aes Branch, OC. 


Development of Evaluators: 
Follow-up e 
to Meeting at Ss | 

my s Locks . eat 


4 to the recent meeting at Chaffey’s Locks, a committee 
of heads of evaluation identified a number of areas for 
discussion that could lead to a shared community agenda. 

e of these was the Development of Evaluators. The 
specific issues suggested were as follows: 


» What roles might the OCG and departments play in 
training and career development to ensure there is a 
sufficient number of qualified staff in the federal program 
evaluation function as it evolves? 


» What are the implications brought on by PS2000, the 

_ SM-EX reduction exercise, and other initiatives for the 

| training and career development of professional 
evaluators? 


Discussion in both the syndicate and the plenary sessions 
was quite animated; the subject was obviously of interest 
nd concern to many. The following summary is based on a 
eport prepared by the discussion leader, Bob Lahey, 
‘orestry Canada. 


’ It is important to recognize two groups within the 
program evaluation community: a “core” group that is 
relatively established and experienced in program 
evaluation and a “transitory” group, some of whom are on 
assignment or secondment from line areas (perhaps for 
one to two years). The function needs both types of 
people as they bring different experiences and knowledge 
to it. Implications for training are apparent, however. 
Training remains an ongoing priority for program 
evaluation managers. 


| The implications for training as a result of high turnover 
are of particular importance for small departments and 
agencies. Because relatively fewer resources are available 
to the evaluation unit, costs associated with the search 
for and training of staff are relatively higher. Being able to 
act quickly to train people is important. 


The formal training of evaluators is still somewhat 
responsive and flexible: the OCG Orientation workshops 
for new entrants, university evening courses, the OCG 
Methods workshops for more experienced people, 
Canadian Evaluation Society professional development 
Sessions, and a variety of other seminars and sessions 
aimed at specific knowledge and skill acquisition. Some 
departments (e.g., ND and CIDA) are developing their 


i 


Program Evaluation 


own training packages so as to be able to inject a 
departmental-specific focus. 


+ Recent government initiatives offer both opportunities 
and challenges for the career development of evaluators. 
On the one hand, it will be necessary to identify and 
prepare for new skill requirements associated with 
PS2000, IMAA and closer linkages between the program 
evaluation and internal audit functions. On the other 
hand, the SM-EX “delayering” may accentuate the 
problem of “career plateauing” and impact on the ability 
of the function to attract and retain highly qualified 
people. Doug Wood, Acting Deputy Comptroller General, 
Evaluation and Audit Branch, outlined the initiatives 
undertaken by the OCG. These included interventions 
with the Personnel Policy Branch, TBS, and assumption 
of representation on the Classification Simplification 
Task Force. 


¢ One alternative to the training of new entrants would be 
the transfer across departments of qualified program 
evaluation staff. 


¢ The OCG, working with departments, is well-positioned 
to address these issues, many of which are unique to the 
program evaluation function. 


The report contains a series of initiatives for further 
consideration. The Evaluation and Audit Branch has 
responded to these community-based initiatives. Progress to 
date on each is as follows. 


Establish a working group from within the program 
evaluation community (departments plus OCG 
representation) to review the whole area of 
training and development including the future 
development of courses, and the role of the OCG, 
CES and departments in the development, funding 
and delivery of training. This group might also 
assist in or take responsibility for initiatives listed 
below and report back to next year’s annual 
meeting of heads of evaluation. 


We have established the Program Evaluation Community 
Development Committee, which consists of both heads 

of evaluation and members of the OCG. This committee 

will provide input to and assist with various initiatives 

in areas of community development (human resource 
planning, training and development, staffing, and 
organization and classification). Thus it is within the domain 
of the committee to examine the entire area of training 

and development either directly itself, or through a 
sub-committee. 


OCG to develop “shopping lists” of both vacancies 
and “interested parties” in the program evaluation 
community willing to transfer across departments 
on either an assignment-secondment or permanent 
basis. 


Program Evaluation 


OCG to look at ways to facilitate the easy transfer 
on assignment from one program evaluation unit to 
another, across departments. People interested in 
transfers or assignments for career-development 
reasons should be encouraged to arrange swaps 
with their counterparts. This would perhaps 
facilitate the process. 


Generally speaking, assignment programs are of greatest 
use in a community in which mobility is low. As was 
discussed at Chaffey’s Locks, the fact that mobility into and 
out of our community is relatively high has implications for 
training and development. It also has implications for the 
nature of an assignment program. 


In response to the community-identified need to provide 
assistance to those who might be affected by recent 
budgetary measures, we are working with heads of 
evaluation to maintain an up-to-date list of current and 
anticipated vacancies. These lists are provided to people 
whose employment status has become uncertain. In the 
longer term, the lists will also be useful for those seeking a 
shorter term assignment or secondment. 


Develop a course calendar for Program Evaluation 
that would identify the spectrum of courses 
available. 


We are currently exploring alternate ways of developing 

and maintaining such an inventory. We have contacted the 
National Capital Chapter of the Canadian Evaluation Society 
in this regard. A key issue is the best way to make the 
inventory available to the community. 


OCG to take leadership to establish, on a semi- 
regular basis, a series of relatively informal 
“armchair discussions” (a.k.a. “brown bag 
lunches”) to share departmental program 
evaluation experiences in dealing with a variety of 
topics of common interest (e.g., specific subject 
matter and major government initiatives such as 
PS2000, IMAA and Special Operating Agencies 
(SOAs). These would be intended as a low-cost 
form of technology and information transfer. 


The first of the proposed series of brown bag lunches was 
hosted October 8, 1991, by Cameron McQueen, on 
secondment from the OCG to Revenue Canada, Customs 
and Excise. The invited guest was Ruth Hubbard, Deputy 
Minister, Revenue Canada, Customs and Excise, who 
presented a series of issues to be considered when looking 
at service to the public in a regulatory department. We are 
more than prepared to facilitate and support additional 
events of this type; please call! = 


Rob Jones 


Dr. Jones is a senior evaluation analyst with the 
Community Development and Communications Division 
of the Evaluation and Audit Branch, OCG. 


The Need for Better 
Performance Measurement 


There is a renewed interest in government in measuring and 
| monitoring the performance of programs. The quality of 
performance information available to and used by the 
“government i is a constant theme of the Auditor General and 
the Public Accounts Committee. Under PS2000, empowered 
‘Managers must have adequate information available on their 
programs and operations in order to improve services to 
Canadians. 


4 “Performance measurement” has left a bad taste in many 
L ouths. The success of implementing performance 
Measurement in the 1970s is generally viewed as mixed at 
best. Yet few disagree with the principles and theory behind 
those efforts. And many would agree that we have learned 
fr om these past attempts. Furthermore, many programs 
have useful measures in place. These measures are used for 
improving program delivery, although the experiences 


gained have perhaps not been widely shared. 


One of the differences — and challenges — today is the impor- 
tance placed on a client’s perspective in defining and 
measuring performance, a viewpoint that requires looking 
outside a program at the impact on Canadians. This focus, 
of course, has been the norm for much program evaluation 
over the past 10 years. In their reports, evaluators and 
auditors have frequently commented on the quality of 
existing performance measurement systems, but perhaps 
have not concerned themselves enough with hands-on help 
in improving such systems. With the experience gained over 
the years, evaluators and auditors are a valuable 
departmental resource when it comes to defining and 
measuring performance. 


With this in mind, the Evaluation and Audit Branch of the 
Office of the Comptroller General, along with departments 
and agencies, is pursuing a number of initiatives in the area 
of performance measurement and monitoring. In most 
cases, the work involves developing discussion papers, 
identifying good practices in an area, and developing 
workshops and seminars for disseminating lessons learned. 
Two committees have been formed, one of heads of 
evaluation and one of heads of audit. The main initiatives 
are listed below. 


Guidance on Developing Performance Indicators — A 
draft paper entitled, “Measuring and Monitoring Program 
Performance and Service to the Public”, has been 
distributed and discussions have been held with a number 
of departments. Work is underway to develop a workshop 
training package that departmental evaluators could use to 
assist managers in their department in developing useful 
performance measures. (Ronald C. Corbeil, 613 - 957-7171) 


A Revised Performance Measurement Policy — The 
existing TB policy of performance measurement is being 
revised in light of the government's service-to-the-public 
thrust. (John Mayne, 613 - 957-7177) 
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Guidance on the Measurement of Client Satisfaction — 
A discussion paper entitled, “Measuring Client Satisfaction”, 
has been distributed for comment and work is underway to 
hold a seminar and develop a workshop. (Mare Lemay, 

613 - 957-7181) 


Setting Service Standards — Another important 
component of service to the public is setting realistic 
standards for service delivery. Standards tell the public the 
minimum quality and level of service they can expect. 
Setting such standards involves consultation with clients, 
knowing the current quality and level of service being 
delivered and knowing what the service organization can 
deliver. The work being done in this area is to examine how 
the measurement of existing service levels and quality can 
best contribute to the setting of service standards. 

(John Mayne, 613 - 957-7177) 


Demonstrating Performance Measurement — An 
important aspect of improving service to the public is being 
able to inform them of the quality and level of service 
actually being provided. As the experience with manage- 
ment representations has shown, this is not as easy a task as 
it might first appear. Performance is a multi-faceted and 
complex concept. Demonstrating good performance in a 
credible manner to a sceptical public is a challenge. Building 
on the ideas behind management representations, the work 
being done in this area is aimed at developing useful 
guidance for managers who want to or need to demonstrate 
to their publics just what service is being provided. 
(Cameron McQueen, 613 - 957-7175) 


Evaluation of Consultative Programs - A significant 
PS2000 thrust is to improve the consultation between 

the government and Canadians. A unit at the Privy Council 
Office is developing guidance in this area. Key to successful 
consultation is knowing if it worked. Our work is aimed 

at developing guidance to evaluators and line managers 

on how to assess such consultative programs. 

(Ibolya Galambos, 613 - 957-7183) 


Monitoring of Total Quality Management (TQM) 
Initiatives — Most of the above components underpin the 
TQM initiative with which a number of departments are 
experimenting as a means of improving service to the public. 
Developments within departments are being monitored 
through the TQM Network to identify innovative approaches 
to improving quality of service and employee quality of 
work-life. Work being done in this area is aimed at 
consolidating this experience and developing guidance for 
managers who seek to inculcate quality values in their 
organizations. (Allison Fader, 613 - 957-7230) 


Guidance on Assessing Performance Measurement 
Systems — What makes a good performance information 
system is better understood now than it was 10 years ago. 
Program evaluators frequently review existing measurement 
approaches as part of an evaluation of a program. Internal 
auditors also address such questions as part of an audit of 
departmental management information systems. Our work 
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is aimed at consolidating this experience and developing 
guidance on assessing and auditing ongoing performance 
information systems. (Jay Morrison, 613 - 957-7040) 


Anyone interested in contributing to the work in a particular 


area is invited to contact the responsible OCG person as 
indicated. We are particularly interested in identifying good 
practices in each area that could be shared with others. 


John Mayne 


Dr. Mayne is Director, Evaluation Policy Division of the 
Evaluation and Audit Branch, OCG. 


Exploring Alternative Modes 
of Intervention: Learning 
from Evaluation 


This article describes an Office of the Comptroller 
General initiative and invites the participation of 
departments and agencies. 


In the 1980s, most OECD countries limited the expansion 
of or actually reduced the size of their public sectors. In 
recent years it is economic circumstances, rather than 
ideology, which are forcing governments to make hard 
choices on resource allocations. The challenge becomes 
that of searching for alternatives which are administra- 
tively feasible, cost-efficient and cost-effective. The 
alternatives must, as well, be consistent with general 
governmental policy and respect fundamental equity 
considerations. 


The evaluation function focuses on finding out what works 
in practice and uses that knowledge to improve programs. 
It can help managers and decision-makers meet the 
challenge of finding appropriate alternatives by better 
sharing the knowledge gained in assessing performance. 
To that end, we are proposing an interdepartmental project 
to share experiences. 


Objective of the Project: Learning from 
Federal Evaluation Experience 


The need for identifying and analyzing more challenging 
and practical alternatives to the status quo was articulated 
in the Government Program Evaluation Plan of 1989 
and was repeated recently in Into the 90s. Identifying 
practical and challenging alternatives is partly a creative 
activity. More importantly, however, it is an exercise in 
understanding the basic instruments available to 
government. This understanding includes studying the 
ways different instruments have been applied in 

practice, and the lessons learned about their effectiveness. 


This note describes a project which we believe can con- | 
tribute to the need for alternatives to the status quo. Its 


describing the instruments available to the government ai 
the different ways they can be used. It is aimed at establis 
ing an improved means for federal evaluators to share 
lessons learned. These lessons result from the research tli 
undertake each year, as well as relevant evaluations | 
undertaken elsewhere. 


Methodological Approach: 

Focusing on Instruments, Not Programs 
Programs are collections of authorities and activities use¢ 
a conscious manner to achieve some objective. Instrumeï 
are those analytically distinct mechanisms available to 
government to implement public policy (to achieve progr) 
objectives). In this sense, instruments are means. They a 
the essential focus for ACTE dealing with cost 
effectiveness. 


A conventional approach has been to classify instruments 
in relation to the purpose of the government objectives | 
(social or economic policy objectives). Such a conceptual 
ization and classification of government activity tends to | 
identify programs with established program instruments | 
(Salamon, p. 25). It tends not to be useful in highlighting | 
alternatives. It also makes it more difficult to apply to one 
program lessons learned in a different area. | 


A different approach, and the one pursued in this project, 
is to conceptualize and classify instruments in relation to | 
the nature of the means available to government. This ! 
classification is independent of purpose. In a sense, this | 
approach describes how a government “intervenes” or 
influences society or the economy. The four categories int 
which instruments are classified according to mode of ! 
intervention are described as follows. 
1) Like any legal entity, the government has the 
power to provide services and produce goods 


(this might be called administrative authority). 
2) The government can use its legislative 

power to regulate activity. | 
3) The government can provide financial support 

or incentives for certain aims. 
4) The government can use its dominant position 

in society to provide leadership, to persuade or 

set direction. 


To implement a specific program objective, governments 
choose a mix of instruments from among these four { 
categories. 


The literature on public administration indicates no 
definitive list of policy instruments or interventions. We fe 
confident, however, that this simple and distinct grouping 
will serve as a good starting point for examining 
programming instruments. 


PE NE ER SEE TT EE EGE DTT IESE SEE TOON NS ET TS EB EE TETE ET LE 


Asecond important aspect of describing the means of 
achieving objectives relates to how the government decides 
to execute chosen instruments. In a sense, this refers to the 

delivery mechanisms of the instruments. As with the 
instruments, we need a preliminary view of the different 
delivery mechanisms. At the moment, no simple and distinct 

approach is apparent. As a point of departure, we are using 
‘the perspective of economics, where there are essentially 
four dimensions: 


e the producer, 
¢ production process; 
e product or services; 
e the consumer. 


From our preliminary analysis, these dimensions seem 
reasonably effective. However, it is likely that some 
‘adjustments of this structure (mode of delivery) will become 
evident with further analysis. More work is required in this 
area and this would be one of the first tasks of the working 
groups. 


The table below illustrates what we are trying to 
demonstrate. To understand programming alternatives, 

itis necessary to examine both dimensions (what the 
government does and where it seeks to aim the 
intervention). However, in a specific case the reasonable set 
of alternatives which a department could consider might 
focus on delivery mechanisms, on instruments, or on a 
particular intersection of the two modes (a cell). That is, 
departments have limits on their mandates and activities. In 
Many cases, they can realistically explore only a limited set 
of alternatives. Interdepartmental, intergovernmental or 
centrally driven reviews are often required if the full range of 
alternatives is to be explored. 


Authorities 


| Crown corporation, 
departments 


Contracts — 
out 


Cost-recovery, 
user-fees 


Delivery Mechanisms by Instruments and Focus 
Legislation” 


Regulations 


_Permits/licences _ 


franchise 


Regulation on 
dangerous goods 


Health and safety 
requirement 


Specific requirement; 
e.g., minimum age 


Program Evaluation 


To illustrate the model, consider a proposed program having 
as an objective “ensuring acceptable housing for low income 
families”. Such a program could be implemented in many 
different ways. For delivery mechanisms, the government 
could produce the requisite housing directly, perhaps 
influence the production process by providing the land, or 
purchase or rent the housing for low-cost leasing by tenants. 
As for intruments, the government could also provide 
financial incentives to producers or consumers. While 
regulation is theoretically available, this might be considered 
to be outside federal jurisdiction. The government might 
seek to persuade by setting national standards, perhaps as a 
complement to using the financial support instrument. 


The environmental sector provides a good example as a 
program actually in place. In this case, the federal 
government uses all the instruments. The government 
provides services such as research aimed at reducing 
production of pollutants. It also provides information to 
consumers so they can reduce their indirect production of 
pollutants. In certain areas, the government regulates by 
setting standards for effluent or for the ambient 
environment. It then enforces these standards. 


The area of regulation and the environment has attracted 

a considerable amount of attention. The Law Reform 
Commission of Canada has spent a considerable amount of 
time and energy to review the strategy for implementation 
of pollution control legislation. They concluded that “The 
1980s require further refinement of the legislation in place, 
more comprehensive use of the framework available, more 
rigorous enforcement of command-penalty mechanisms, 
more widespread, open and consistent use of control 


Persuasion/ 
Policy Directions 


Market information, 
association support 


Grant, 
loan — 


ce 


Technology 
information 


Market 
information 


Vouchers Campaign, 


support association 


Program Evaluation 


regimes and incentives, and willingness to try imaginative 
solutions”. At the same time, there is a growing body of 
literature on market-type mechanisms for pollution control 
that provides for more imaginative solutions involving 
different regulatory frameworks. While there is general 
agreement on the necessity of legislation, there is not 
agreement on how legislation should be implemented 

(i.e. the delivery mode). 


These examples are provided only to illustrate the analytical 
examination of alternatives which might be used to obtain 
policy objectives. Moreover, it is evident that other 
dimensions could be added. For example, where the 
government is directly producing services, there are whole 
ranges of other choices in categories, such as technical 
options, organizational alternatives, and even decisions in 
the area of administrative control. 


It is also important to consider some of the factors that will 
be relevant in identifying and analyzing alternatives. In 
addition to effectiveness in achieving the objectives, 
relevant factors to consider are legal authority, international 
conventions, harmonization with related federal and 
provincial programs, equity (from a number of 
perspectives), administrative feasibility and, of course, 

the political preferences of the current government. 


A Call to Establish Working Groups 


The specific tasks of this project are to test and modify 
the approach through further analysis and consultation 
with evaluators and others. Actually using the model as 
an instrument to summarize and communicate useful 
lessons learned from evaluation will lead to modifications 
as well. 


Developing and testing the approach can only be done 
effectively with the cooperation and participation of federal 
evaluators. The federal evaluation community is both the 
principal storehouse of experience on examining programs 
and a potential user of such models and associated lessons 
learned. 


In view of this, we initially plan to establish working groups 
for each of the instruments: administrative authority, 
legislation or regulation, financial support and persuasion or 
policy directions. Each working group will be asked to 
address questions such as: 


e What kinds of federal activity fall within each 
instrument, and can they be well described by the 
mode of delivery? 

+ How widespread is its use and what accounts for the 
pattern of usage that is apparent? 

e Are other approaches to delivery within the instruments 
considered as potential alternatives? 
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e What are the major tasks that must be carried out and 
who is responsible for them? 


e What are the principal lessons learned regarding the 
instruments or aspects of its delivery? 


It is anticipated that, in order to share experiences broadly, 
papers will be prepared on each instrument. These paper 
topics would include a conceptual framework, some lesso 
learned, and perhaps a strategy for continuing work. In 
addition, we hope members of the evaluation community 
will help provide an annotated bibliography of relevant 
materials from other sources. In this regard, we are 
monitoring the OECD project on Market-Type Mechanisms 
that was described in an earlier newsletter. 


We are interested in your participation. Involvement may — 
range from providing advice to leading particular working 
groups. If you are interested in participating or have 
questions or suggestions, please contact Donald Lemaire 
at (613) 957-7176. | 


Note: A short bibliography on the subject matter is 
provided for your information. 


Linder, Stephen H. and B. Guy Peters. “Instruments of 
Government: Perceptions and Contexts”. Journal of Public 
Policy, Vol. 9, No. 1 (1989). 


Linder, Stephen H. and B. Guy Peters. “From Social Theory 
to Policy Design.” Journal of Public Policy, Vol. 4, No. 3 
(1984). : 


May, Peter J. “Hints for Crafting Alternative Policies”. 
Policy Analysis, Vol. 7, No. 2 (1981). 


Auld, D. and H. Kitchen. “The Supply of Government 
Services”. The Fraser Institute, 1988. 


Law Reform Commission of Canada. “Crimes Against the 
Environment”, Working Paper 44, 1985. 


Law Reform Commission of Canada. “Policy 
Implementation, Compliance and Administrative Law”, 
Working Paper 51, 1986. 

| 
Salamon, Lester M. ed. “Beyond Privatization: The Tools … 
of Government Action”. Washington, D.C.: The Urban 
Institute, 1989. 


Donald Lemair 


Mr. Lemaire is an evaluation analyst in the Operations 
Division of the Evaluation and Audit Branch, OCG. 
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Capital facilities and Community Services Activity / Activité des 
Immobilisations et services communautaires : 
($592.1 million(s) & 121 PYs/AP) 


e Capital Program: Allocation Mechanism // 
Programme d’allocation des fonds d’immobilisations 


Economic Development Activity / Activité de développement 

économique : 

($92.7 million(s) & 275 PYs/AP) 

e Systematic Approach for Evaluating Economic 
Development Programs // Une approche systématique 
pour l’évaluation des programmes en développement 
économique (résumé à l'intention de la gestion) 


e Saskatchewan Indian Agricultural Program // 
Programme agricole des Indiens de la Saskatchewan 
(résumé à l'intention de la gestion) 


+ Manitoba Indian Agricultural Program // Programme 
agricole des Indiens du Manitoba (résumé à 
l'intention de la gestion) 


Economic Development and Resources Management Activity / Activité 

de développement économique et de gestion des ressources : 

($72.1 million(s) and 439 PYs/AP) 

e Yukon Northern Land Use Planning // Planification 
d'aménagement du territoire — Yukon (résumé à 
l'intention de la gestion) 


e N.W.T. Land Use Planning // Planification 
d'aménagement du territoire — T.N.-O. (résumé à 
l'intention de la gestion) 


e Canada N.W.T. EDA — Phase J and II 


e Development Impact Zone Groups // Groupes 
régionaux d’exploitation pétroliére et gaziére 
(résumé à l'intention de la gestion) 


Education Activity / Activité d'enseignement : 
($735.9 million(s) & 743 PYs/AP) 


e Cultural/Educational Centres Program // 
Programmes des centres éducatifs/culturels 


e Post-Secondary Education Information System // 
Système d’information de gestion du programme 
d’aide à l'éducation post-secondaire (résumé à 
l'intention de la gestion) 

e Anishinaabe Child and Family Services Training 
Program // Programme de formation pour les services 
à l’enfance et à la famille Anishinaabe (résumé à 
l'intention de la gestion) 

e Post-Secondary Education Assistance Program // 
Programme d’aide à l'éducation post-secondaire 
(résumé à l'intention de la gestion) 

e Primary and Secondary Schools (Evaluation Guide) // 


Guide d’évaluation des écoles primaires et 
secondaires 


° Management Training for Band Leaders and 
Managers // Formation en gestion offerte aux 
dirigeants et aux gestionnaires de bandes 


Lands, Revenues and Trusts Activity / Activité des terres, revenus 
et fidéicommis : 
($58.3 million(s) & 555 PYs/AP) 


e Test Case Funding Program // Programme de 
financement des causes types (résumé à l'intention 
de la gestion) 


e Cree-Naskapi Commission 


e Summary Report: Impacts of the 1985 Amendments 
to the Indian Act (Bill C-31) // Rapport sommaire: 
Répercussions de 1985 à la Loi sur les Indiens 
(Projet de Loi C-31) 


e Research Funding Division 


Self-Government Activity / Activité de l'autonomie gouvernementale : 
($17.2 million(s) & 83 PYs/AP) 


° Self-Government Negotiations Funding // 
Négociations de l’autonomie gouvernementale 
(résumé à l'intention de la gestion) 


¢ Indian Commission of Ontario 


Industry, Science and Technology Canada 
Industrie, Sciences et Technologie Canada 


Aboriginal Economic Program Activity / Activité du programme 
de développement économique des autochtones 
($78.8 million(s) & 90 PYs/AP) 


¢ Alberta Northern Development ERDA 
(Mid-Term Evaluation) 


Industry and Technology Development Activity / Activité de 
développement de l'industrie et de la technologie : 

($444.1 million(s) & 650 PYs/AP) 

¢ British Columbia Science and Technology ERDA 
° Manitoba Urban Business Agreement ERDA 

° Nova Scotia Tourism Development ERDA 


e Prince Edward Island Tourism Development ERDA 


Regional Development Québec and Ontario Activity / Activité de 
développement régional au Québec et en Ontario : 

($201.6 million(s) & 232 PYs/AP) 

e Industrial and Regional Development Program 


° Ontario Tornado Damage Business Assistance 


Justice Canada 


Administration Activity / Activité d'administration : 
($22.1 million(s) & 262 PYs/AP) 


© Central Divorce Registry 


Legal Policy & Program Development Activity / Activité d'élaboration 

de la Politique et des programmes en matière Juridique : 

($256.9 million(s) & 158 PYs/AP) 

° Divorce Act // Loi sur le divorce 

° Legal Aid: Manitoba — Alternatives: Financial 
Eligibility Criteria 

¢ Legal Aid: Manitoba — Policy: Financial Eligibility 
Criteria 

° Public Legal Education and Information // 
Vulgarisation et information juridiques 


Litigation Services Activity / Activité des services des contentieux : 
($62.9 million(s) & 686 PYs/APs) 


e Litigation Services // Services des contentieux 


Labour Canada 

Travail Canada 

Labour Operations Activity / Activité des opérations du travail : 
($38.4 million(s) & 461 PYs/AP) 

e Labour Education — 2nd Study 

e Labour Standards 


© Occupational Safety & Health 


National Defence 

Defense nationale 

Policy Direction & Management Services Activity / Activité de direction 

des politiques et services de gestion : 

($674.4 million(s) & 2,173 PYs/AP) 

° Geographic Support (Mapping and Charting) // 
L’aide géographique (résumé à l'intention de 
la gestion) 


National Energy Board 
Office national de l'énergie 


Energy Regulation & Advice Activity / Activité de réglementation 
et consultation en matière d'énergie : 
($25.6 million(s) & 336 PYs/AP) 


e Energy Supply 
e Regulatory Support Function: Office of the Secretary 


National Parole Board 

Commission nationale des libérations conditionnelles 

Parole Board Operations Activity / Activité des opérations de 

la Commission : 

($22.0 million(s) & 284 PYs/AP) 

e Detention Provisions of Bill C-67 (jointly with 
Correctional Service Canada) // Dispositions du 
projet de loi relatives au maintien en incarcération 
(conjointement avec Service correctionnel Canada) 


National Research Council of Canada 

Conseil national de recherches Canada 

National Research and Development Activity / Activité de recherche 

et développement dans l'intérêt national : 

($271.3 million(s) & 1,692 PYs/AP) 

e Atlantic Research Laboratory // Laboratoire de 
recherche de l’Atlantique (résumé à l'intention 
de la gestion) 


e Review Committee on the Atlantic Research 
Laboratory // Comité de révision du laboratoire 
de recherche de l'Atlantique 


Natural Sciences and Engineering Research Council 
of Canada 

Conseil de recherches en sciences naturelles et 

en génie du Canada 


Grants and Scholarships Activity / Activité des subventions 
et bourses : 
($406.7 million(s)) 


e Equipment Grants 


Public Service Commission of Canada 

Commission de la fonction publique du Canada 

Staffing Programs Activity / Activité des programmes de dotation : 
($63.8 million(s) & 1,097 PYs/AP) 

e Career Assignment Program 


e Impact on the Staffing Process of Publicizing Vacancy 
Notices for Management Category Positions 


e A Review of Post-Secondary Recruitment in the 
Federal Public Service 


Training Programs Activity / Activité des programmes de formation : 
($34.8 million(s) & 673 PYs/AP) 
e The Role of the Public Service Commission in 
Providing Professional, Managerial and 
Technical Training 


Revenue Canada, Customs and Excise 
Revenu Canada, Douanes et Accise 


Corporate Administration Activity / Activité d'administration 
ministérielle : 
($73.4 million(s) & 802 PYs/AP) 


e Administrative Reform of Staffing 
Customs Activity / Activité des douanes : 
($383.9 million(s) & 7,390 PYs/AP) 


° Common Border Facilities // Installations frontalières 
communes 


Revenue Canada, Taxation 
Revenu Canada, Impôt 


Appeals Activity / Activité des avis d'opposition et appels : 
($36.2 million(s) & 678 PYs/AP) 


+ Appeals Program // Programme des appels 


Post-Assessing Compliance Programs Activi 

1 Acth 
vérification des déclarations d'impôt des ne an = 
($464.6 million(s) & 8,330 PYs/AP) ; 


Office Examination Program // Programme d’examen 
du Bureau 


Assistance to Tax] 

d'elde aux contribuables et établissements des nanas sume 

($339.9 million(s) & 7,749 PYs/AP) À 

° Telephone Enquiries Program; Report on the 
Quality of Service // Programme des demandes 
de renseignements par téléphone; rapport sur la 
qualité du service 


Secretary of State 

Secrétariat d'Etat 

Social Development Activity / Activité de dével : 

($71.2 million(s) & 107 PYSAP) Be nee oe 

e Disabled Persons Participation Program // 
Programme de participation des personnes 
handicapées (résumé à l'intention de la gestion) 


Social Sciences and Humanities Research Council 
of Canada 
Conseil de recherches en sciences humaines du Canada 


Grants and Scholarships Activity / Activité des subventions 
et bourses : 


($82.4 million(s)) 
e Aid to Scholarly Publications Programme 


Statistics Canada 
Statistique Canada 


Census and Social Statistics Activity / Activité du recensement et 

des statistiques sociales : 

($61.1 million(s) & 371 PYS/AP) 

° Target Groups, Canadian Social Trends and the 
General Social Survey: Executive Summary, 
Recommendations and Field Response // Groupes 
cibles, tendances sociales canadiennes et enquéte 
sociale générale: résumé, recommandations et 
réactions du secteur 


Institution Statistics Activity / Activité des statistiques des institutions : 
($18.7 million(s) & 300 PYs/AP) 


° Culture Statistics Program 


International and Domestic Economic Statistics Activity / Activité de 
la statistique économique nationale et internationale : 
($73.8 million(s) & 1,220 PYs/AP) 


° Manufacturing Statistics Program 
e Energy Statistics Program 
e Construction Statistics 


° Investment and Capital Stock Program // Programme 
des investissements et des stocks de capital 


Socio-Economic Statistics Activity / Activité de la statistique socio- 
économique: 
($41.5 million(s) & 586 PYs/AP) 


e Statistics Canada Labour Statistics Program 


e Goods and Services Producing and Capital 
Expenditures Price Indexes // Indices des prix des 
industries de biens et services et des immobilisations 


Transport Canada 

Transports Canada 

Aviation Activity / Activité de aviation : 
($437.4 million(s) & 6,904 PYs/AP) 


e Aviation Regulation and Safety Programs 


Marine/Canadian Coast Guard Activity / Activité de la Marine/Garde 
côtière canadienne : 
($722.3 million(s) & 5,866 PYs/APs) 


e Marine Environmental Protection and Clean-up 


Policy and Coordination Activity / Activité des politiques et de 
coordination ($883.9 million(s) & 297 PYs/AP) & Surface Activity / 
Activité de surface ($62.0 million(s) & 418 PYS/AP) 


e Branch Line Abandonment 
e Montreal Commuter Rail 


Veterans Affairs Canada 
Anciens Combattants Canada 


Departmental Administration Activity / Activité d'administration 
du Ministère : 


($31.1 million(s) & 427 PYs/AP) 

e Initial Portfolio Response to Report on Barriers to 
Women in the Public Service // Réponse initiale du 
portefeuille au Rapport sur les obstacles rencontrés 
par les femmes dans la fonction publique 


Economic Support Activity / Activité de soutien financier : 

($260.0 million(s) & 362 PYS/AP) 

e Assistance Fund // Programme du fonds de secours 
(résumé à l'intention de la gestion) 


Health Care Activity / Activité des soins de santé : 

($491.4 million(s) & 2,233 PYs/AP) 

¢ Institutional Care Program // Programme des soins 
en établissements (résumé à l'intention de la gestion) 


Western Economic Diversification Canada 
Diversification de l'économie de l'Ouest Canada 


Western Economic Diversification Activity / Activité de diversification 
de l'économie de l'Ouest : 
($286.0 million(s) & 313 PYs/AP) 


e Airshow Canada 


¢ WED Client Satisfaction // Satisfaction des clients 
de DEO 
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strategic Analysis in Program Evaluation: 


Means of incorporating the political, administrative 
and social elements of government PIRE 


in the evaluation process! 


Evaluators are often accused of living in 
a theoretical world that does not take 
into consideration the reality in which 
decisions are made about the design 
and modification of programs. Judging 
by the limited use that the “real 
decision-makers” — members of Cabinet, 
senior officials and central agencies — 
make of evaluation findings, there is 
probably some truth to this accusation. 


Part of the problem stems from the fact 
that evaluation is approached from a 
point of view that, in the name of 
objectivity and methodological 
precision, tends to take the program out 
of its environment and to assess it 
“coldly”, using essentially technical 
criteria such as relevance, impact, 
effectiveness, efficiency and economy. Of 


1 The opinions expressed in this article are those of 
the author and do not represent the position of the 
Office of the Comptroller General or the 
Government of Canada. 


course, the purpose of this approach 

is to try to maximize program 
performance. At first glance, the 
objective is praiseworthy in itself; 
shouldn't public decision-makers try to 
maximize the way they use taxpayers’ 
money? Clearly, the answer is 
affirmative. But that is not the 
question. What should be asked is 
whether the objectives of relevance, 
effectiveness, efficiency and economy 
do, in fact, provide an assessment of the 
government action and represent the 
determining factors in the decision- 
making process. 


In a perfectly rational world, that of the 
technocratic ideal, the answers to these 
questions would also be in the 
affirmative. Everyone will agree, 
however, that the world of public 
administration is not one of perfect 
rationality. Many government programs 
— perhaps even all government 
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programs — appear to be more a compromise than the 
optimum solution to a given problem. Rarely is a program 
the result of a unilateral and enlightened decision of an 
omnipotent and omniscient decision-maker(!). Program 
“decisions” are made through consultation and even 
negotiations in which various parties, such as central 
agencies, the other departments involved, provincial 
governments, program staff and interest or user groups, 
all exert some degree of influence on the final outcome. 
The option retained will very likely not be the best possible 
solution but rather, as the eminent sociologist Herbert 
Simon put it, the first option that appears satisfactory to 
the key parties. 


To evaluate government programs as though they were the 
best possible solution to a problem, and essentially fulfilled 
the imperatives of effectiveness, efficiency and economy, 
goes against the intrinsic nature of programs. Such an 
approach, therefore, creates a problem. One result may be 
conclusions and recommendations that are “out of touch” 
with the specific context in which the program operates and 
was designed. A solution to this situation, which also 
silences those who criticize evaluation for its theoretical 
approach and lack of practical impact, is to reintroduce to 
the evaluation process the political, administrative and 
social environment from which the government program 
emanates. 


Strategic analysis is a tool that makes it possible to take 
the environment into consideration. It is based on the 
following two assumptions: (1) all phenomena have meaning 
and reflect a certain rationality from the moment they exist; 
and (2) the outcome of organized action (for example, a 
government program) is always less “rational” than the 
action of the individual actors from which it originated. 
Consequently, the apparent irrationality of a phenomenon, 
such as the continuation of a program after the original 
problem has disappeared, could be explained by behaviours 
that, from the perspective of the actors (i.e., program 
officials), are completely rational. 


Strategic analysis is an inductive research activity and is 
essentially empirical. It is intended to explain what has 
happened (the action system), rather than what should have 
happened (the program model). It is an approach that 
focuses on the “actors” and tries to explain their motivations 
and their influence and to understand the actions they take. 
It involves collecting data to provide the evaluator with very 
concrete information on how the individuals or groups 
involved in the program perceive and experience their 
respective situations: how do they assess their relationships 
with other actors; how do they conduct their activities; how 
do they perceive their options and those of other actors; 
what are the major problems, conflicts or constraints that 
they faced; how are these problems or conflicts usually 
resolved, etc. This information could take the form of 
qualitative data obtained through a series of informal 


interviews or quantitative data collected from a statisti 
analysis of answers to a formal questionnaire. The evalua’ 
compares the responses of every group of actors to 
reconstruct the program’s “social system”, to understand 
strategies of the different actors, as well as to identify the 
“informal” rules that govern interaction between the vari 
parties involved in the program, affect their actions and 
impact on the program’s performance and future 
development?. In this way, strategic analysis makes it 
possible to take into consideration an incontrovertible 
reality: government programs are established, managed 
and used by people; they do not obey physical or 
mechanical laws. 


For evaluators, strategic analysis is beneficial in that it 
gives proper weight to the roles, interests and influence of 
the various program stakeholders. It also facilitates an 
understanding of the fundamental goals of these 

stakeholders and makes it possible to determine who | 
exercises what type of control on the performance and futu 
development of the program. This information is very 
important because it helps evaluators determine to what | 
extent and in what ways changes may be made to the 
program to improve its performance. It also allows | 
evaluators to be more on the “same wavelength” as their | 
clients. Strategic analysis places evaluators “in the clients’ 
shoes”, and allows them to understand their clients’ aims | 
and the constraints within which clients must work. As a | 
result, the information presented to clients is more likely 
be better adapted to their decision-making environment. 


More specifically, strategic analysis makes it possible: | 


with respect to the presentation of the evaluation! 
findings, to go beyond a simple statement and to provic 
a degree of explanation. The evaluation should not 
merely highlight badly-defined program objectives, the | 
poor “targeting” of the clientele, the continuance of the 
program in spite of the disappearance of the original 
problem, or even the general satisfaction of the clientele 
in spite of the “objective” ineffectiveness of the program. 
It must also try to understand why things are as they © 
are. By identifying the causes of a problem — which are” 
not simply a set of statistical correlations — the evaluato 
has already gone a long way to finding realistic solution: 
By focusing on the actions, influence and goals of the 
various stakeholders, the strategic analysis can clarify | 
situations that, seen Gan the outside, appear odd; | 
| 
# 
| 


with respect to the conclusions, to assess the 
relevance, impact, effectiveness, efficiency and economy | 
of the program, while giving ie consideration to the 
political, administrative, financial and other constraints 


2 The principles of strategic analysis are described in detail in the classic work ¢ 
Michel Crozier and Erhard Friedberg, L'acteur et le système, Paris, Éditions d 
Seuil, 1977. 
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… that affect the program and prevent its maximum 

- performance. In this way, the overall assessment of the 
program’s performance will not have as its starting point 

a theoretical model of “optimum” performance, but rather 

the maximum level of performance that the program can 

reasonably achieve given the constraining context in 

which it develops. This setting can only serve to increase 

- the credibility and legitimacy of the evaluation’s 
conclusions; and 


with respect to the recommendations, to propose 
solutions that display a greater political sensitivity. In 
other words, to formulate recommendations that not only 
reflect departmental and government priorities, but that 
are also politically and administratively feasible. In this 
sense, strategic analysis can significantly enhance the 
practical impact of an evaluation. By shedding light on 
the goals and behaviour of the various parties, it can 
identify the options available to program officials. It 
makes it possible to present recommendations that take 
these options into consideration and that do not simply 
list the changes that have to be made to the programs. 
Instead, the recommendations can specify how the 
changes should be made in order to be successful. 


jome “purists” may view this evaluation approach as 

leresy It may, indeed, look quite revolutionary. It is in 
irect conflict with the “traditional” concept of evaluation 
hat requires evaluators, for the sake of objectivity, to 
naintain some distance from the object being analyzed- 

he government program. An evaluation that incorporates 
trategic analysis assumes, on the other hand, that 
valuators get as close as possible to the “actors” in the 
rogram — and even that they put themselves in their 

ace — in order to understand their motivation, their 
tions and their impact on the program’s performance 

nd future direction. Certainly, this is an evaluation 
nethodology that is oriented more toward action, and that 
ncorporates the social system inherent in each government 
rogram — rather than denying it. Given the somewhat 
imited success of “traditional” evaluations in terms of their 
mpact on decision-makers, there is probably room for 
trategic analysis in program evaluations, particularly at 
time when it is vitally important to make the discipline 
more productive”. 


Nould the introduction of strategic analysis to program 
valuation make the evaluation less critical, more 
omplaisant, and prevent it from being a true review of 
rograms? Personally, I do not think so. The primary 
bjective of the evaluation remains the same: to analyze the 
erformance of programs and to propose solutions to 

mprove that performance where necessary. Moreover, there 
nust be the same rigorous application of the methodological 
ools. The major distinction is primarily in analyzing and 
nterpreting the information that is collected. It might even 
e possible to argue that this evaluation methodology is the 
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best guarantee against complaisant evaluations. There is no 
longer a need to convince clients of the benefits of evaluation 
by presenting the information they want to hear. Rather, the 
benefits of the evaluation can be shown by providing 
information that is “linked” to what clients “experience”, 
and that provides him with practical solutions to the real 
problems with which they are faced in the decision-making 
process. An evaluation process that gives due consideration 
to the context in which government programs have been 
designed and operate, as well as to the constraints limiting 
the program managers’ margin for manoeuvring is not a 
process that tries to excuse poor performance; it is 
essentially a realistic process. 


If we want program evaluation to have an impact where it 
counts, that is, with members of Cabinet, senior officials and 
central agencies, it will be necessary to give them a strategic 
decision-making tool that reflects and is adapted to their 
reality, rather than a tool that tries to change their reality or 
that considers the technocratic ideal as the reality. Strategic 
analysis is a method that should enable program evaluation 
to move in that direction, without transforming the role of 
the evaluator into that of a management consultant. 


Marc Lemay 


Mr. Lemay is an evaluation policy analyst with the 
Evaluation Policy Division of the Evaluation and Audit 
Branch, OCG. 


PS: Please feel free to contact me at (613) 957-7181 if you 
have any comments or if you want to know more about 
strategic analysis. 


New Publication: 
R&D Evaluation 
Canada 


Pursuing its shared community agenda, the OCG is 
sponsoring a newsletter on the evaluation of research and 
development in Canada. The publication will be written and 
edited by practicing R&D evaluators-managers and will be 
distributed free of charge. 


The basic objective of the publication is to develop linkages 
between R&D evaluators and managers to facilitate the 
exchange of ideas and experiences. Evaluation of R&D 
presents unique challenges and problems that are not 
always addressed in works on program evaluation in 
general. The purpose of the newsletter is to help bridge the 
information gap with summaries of work completed, notes 
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on methodology, pointers to useful publications, lessons 
learned and best practices. To ensure that a variety of 
perspectives is presented, contributions will be solicited 
from program evaluators, auditors, researchers and science 
and technology managers in universities, consulting groups, 
industry and government. 


If you would like to be on the mailing list for “Evaluation of 
R&D”, please write to the Associate Editor, 10th Floor West 
Tower, L'Esplanade Laurier, Ottawa, K1A 1E4 (or FAX (613) 
957-7240), indicating your complete mailing address, 
telephone and fax numbers, area of specialization and, if 
relevant, your willingness to contribute material. 


Michael O’Brecht 

Program Evaluation 

Medical Research Council 
of Canada 


Mr. O’Brecht is the founding editor of R&D Evaluation 
Canada. 


Students in Master’s in Public 
Administration Program 
seeking Practical Training 
Assignments 


The Ecole nationale d’administration publique (ENAP) 
recently introduced a new Master’s in Public 
Administration program for young undergraduates and 
persons already working in this field but who are interested 
in improving their knowledge and skills. The purpose of this 
program is to train analysts, advisors or researchers in the 
specialized fields of public administration, such as 
organizational analysis and development, human resources 
management, program evaluation, and accounting and 
financial management. In offering this program, ENAP 
hopes to provide employers with candidates who: 


¢ know and understand the phenomena that affect the 
public administration environment, its constraints and 
its dynamics; 


* know and can apply research methods and techniques 
introducing change to organizations; 


¢ know and can use theories, models and specific analysis 
frameworks in their field of specialization; 


* know the current policies and practices of public 
organizations and are able to make a critical analysis 
ofthem, to identify problems with them and to suggest 
appropriate solutions; and 


* have the skills required for their role as advisors, _ 
especially those skills which will enable them to # 
establish a relationship of trust with their “client” and 
to communicate effectively the results of their analyses 
and research. 


The training that is offered by ÉNAP includes a 15-week 
practical training assignment in a public organization 
that students can obtain concrete work experience in then 
field of specialization. 


As the coordinator of the practical training assignments, . 
am seeking your cooperation in offering assignments tc 
ENAP students in program evaluation or in any of thi 
above-mentioned disciplines for September 1992. Please fe 
free to contact me for further information. 


Mrs. Maria Dame 
Coordonateur des stages | 
ENAP (Montreal) _ | 
(514) 522-3641, ext. 556 | 
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Total Quality Management and 
he Review Functions 


Irganizations worldwide have embarked on a period of soul- 
searching, redefinition and reconstruction in an atmosphere 
sharged by global competition. Public sector organizations 
1ave also had to respond to the changing environment, with 
ts conflicting demands. Organizations have adopted general 
strategies for renewal, focusing on their service mission and 
he human capital that is the driving force behind 
accomplishing that mandate. Total quality management is 
me such strategy. 


What is Total Quality Management? 


lotal quality management (TQM) is defined by the U. S. 
‘ederal Quality Institute as “a strategic, integrated 
nanagement system for achieving customer satisfaction 
which involves all managers and employees and uses 
juantitative methods to continuously improve an 
rganization’s effectiveness”. Its cornerstones are: 
ontinuous improvement in the way an organization 
unctions, the involvement of all employees in the change 
rocess and striving for customer satisfaction. Total quality 
nanagement has evolved from the statistical process control 
ind quality assurance concepts of such quality gurus as 
Jeming, Juran and Crosby. It focuses not only on doing 
hings right the first time, but on doing the right things. 


[QM initiatives encompass all the processes of the 
rganization, drawing on management, employees, 
ustomers and suppliers to identify improvement 
pportunities and solutions. It is both strategic — quality is a 
ompetitive strategy for long-term success — and systematic 
-all production and administrative systems are aligned 
vith the organization’s mission and values. Other key 
ements include: effective communication and recognition, 
eliance on standards and measures, and teamwork and 
raining. It is noteworthy that the foregoing match almost 
xactly the seven attributes of productivity management in 
brecent draft study by the Ottawa Chapter of the Institute 
f Internal Auditors. These attributes are: focus on the 
ustomer; long-term commitment; senior management 
upport and direction; emphasized employee participation; 
tandards, measures and feedback; commitment to training; 
ind rewards and recognition. Some similarities to the 
yanadian Comprehensive Auditing Foundation’s 12 
ttributes of effectiveness are also evident, notwithstanding 
he differences in purposes and applied methodologies 
mong the various approaches to improving management. 
in excellent reference to which you might wish to refer for 
nore in-depth information is Excellence in Government: 
otal Quality Management in the 1990s by David Carr and 
an Littman. 


Program Evaluation 


Why TQM? 


In the private sector, the answer is probably self-evident. 
TQM’s goal is to improve the quality of products and 
services, while reducing cost, delay and inefficiency. TQM 
has been shown, in a recent U.S. study in the private sector, 
to increase productivity, market share, customer satisfaction 
and profitability. 


In the public sector, where we are not driven by profit, the 


answer may not be quite as clear-cut. Deficit reduction, 


productivity and better service may be motivators, but the 
“customers” of government services may have no alternative 
choices, may be difficult to define and frequently have 
widely diverging and conflicting needs and desires. Indeed, 
the same customer may have intrinsic conflicts of interest, 
depending on which role he or she is cast in at a given 
moment. How can one reconcile the various interests of 
those affected by a regulatory program, which may include 
taxpayers, corporate regulatees and their shareholders, 
and other levels of government? How can a manager 
respond to governmental and public pressures to improve 
service while facing the constraints of increasingly 

limited resources? 


There are no “pat answers”. However, the elimination of 
processes which add no value to government programming 
is in line with TQM principles and is vital to creating or 
enhancing excellent or well-performing organizations that 
are able to do more (or the same) with less. A sense of pride 
in one’s work and the organization’s accomplishments, along 
with an enhanced image and respect for the Public Service, 
can improve employee morale, while also supporting the 
organization’s survival and competitiveness for optional 
customers and a skilled labour force. 


TQM in the Public Sector 


Right now, more and more government organizations are 
implementing TQM initiatives at a variety of levels. The 
provinces of Ontario, Saskatchewan and, in particular, 
British Columbia, are leading the way in quality 
improvement projects. “Service Quality B.C.!” is how the 
telephone gets answered in a fledgling organization in 
Victoria. This is where Brian Marson, B.C.’s Comptroller 
General, is temporarily hanging his hat while he serves as 
Chairman of the Service Quality B.C. Secretariat. In the fall 
of 1990, the Secretariat launched its ambitious program of 
improving service to the public, reducing red tape, 
delegating authority and promoting management 
excellence. Since then, all provincial Public Service 
employees have received the Secretariat’s booklet, “Service 
Quality — an Introduction”; each ministry has appointed a 
service quality leader and has developed training plans to 
improve customer service; over 3,000 employees have 
participated in regional consultations, service quality 
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training and showcases of service quality success stories; 
and most ministries have developed service quality 
action plans. 


On the federal scene, here in Ottawa, departments involved 
in delivering vastly differing services to a plethora of 
“customers” are actively pursuing TQM initiatives. The 
Department of National Defence started its TQM efforts 
two years ago and anticipates that the initial process will 
take five to seven years. Its thrust of assuring quality in 
purchased military materiel was spurred on by its interface 
with its American counterpart, resulting in executive 
training, awareness material and briefings, Quality Councils 
and initial TQM projects. Industry, Science and Technology 
Canada has recently conducted the department's self- 
assessment against the criteria developed for the Malcolm 
Baldridge National Quality Award — the U.S. Department of 
Commerces prestigious award for excellence in the private 
sector. This information served as a baseline measure and is 
assisting in determining the direction of TQM efforts. Health 
and Welfare Canada’s Income Security Program is 
undergoing an extensive redesign project, based on TQM 
principles, working in partnership with its Personnel 
Administration Branch. Transport Canada’s Training 
Directorate embarked upon a formal quality improvement 
process three years ago using a values-oriented approach, 
which has been fairly successful in moving toward a more 
participative, mutually respectful and client-focused culture. 
Energy, Mines and Resources has embarked upon 
“Excellence EMR”, with its Mineral and Engineering 
Technology Sector (METS) being the most actively involved 
(currently developing its action plan). Supply and Services 
Canada, the Office of the Auditor General, the National 
Capital Commission, Western Economic Diversification and 
Indian and Northern Affairs Canada are also in the 
preliminary stages of TQM initiatives. 


A network of people involved in TQM efforts in the federal 
government meet monthly to exchange ideas, knowledge 
and experiences, with the aim of furthering the 
implementation of TQM within the public sector. An offshoot 
of this network is currently preparing a proposal to the 
Human Resources Development Council concerning the 
establishment of an award for quality improvement in the 
federal public sector, possibly similar to the President’s 
Award for Quality and Productivity Improvement in the 
United States. A series of modular workshops on TQM- 
related topics is also being organized, with the goal of setting 
up a national conference on TQM, possibly in the fall of 
1992. (For further information, contact the author or Larry 
Cassie at 952-3100.) 


How is TQM Implemented? 


Improvement opportunities may be spotted by assessing 
organization against well-defined standards of 
organizational excellence, such as those of the Malcolm 
Baldridge Award. Other useful tools for highlighting 
problem areas include: flow charts, histograms, Pareto ~ 
charts, scatter charts and cause-and-effect diagrams. 
Improvement procedures tend to follow four basic steps: 
plan what you're going to do, do a pilot project-experiment, 
check its results and act on them. TQM may be initiated 
concurrently with a mission statement development exerci | 
or may arise out of one. Experience has shown that senior) 
management’s commitment is crucial to success and that | 
solid preparation and management is required. Also 
important is the putting into place of a proper 
infrastructure, such as a corporate steering committee. 
Essential as well are proper research — assessment or A! 
measurement of current performance, using employees anc 
both internal and external customers — and a clear 
definition of roles and responsibilities between facilitators | 
and line managers. Comparable organizations or | 
established awards criteria, e.g., Canadian Awards for 
Business Excellence, may serve as a comparison point or | 
benchmark for the organization’s analysis. Objectives must 
be set and the change process managed. Progress must be 
measured, reported on and celebrated. 


What Role Can Internal Audit and 
Program Evaluation Play? 


Organizations that are dedicated to excellence exhibit a 
number of common characteristics, which have been 
incorporated into the criteria for a number of awards for 
excellence. Well-developed systems of measurement and 
analysis that demonstrate quality and productivity 
improvements, along with quality assurance assessments 0 
products or services and customer satisfaction, are all | 
characteristics of organizations devoted to excellence. 
Evaluation, in focusing on results, and audit, in focusing on 
internal processes, have appropriate orientations and 
essential technical skills to assist in quality improvement | 
initiatives. Their focus on productivity, processes, client 
satisfaction assessment and the measurement of service 
quality is key to the continuous improvement process. 


| 
| 


The assistance of audit or evaluation in the development of | 
a few critical measurements by management and staff, | 
whether quantitative (e.g., the percentage of time in which | 
the turnaround targets are met) or qualitative (e.g., the 
client satisfaction with the courtesy and helpfulness of | 
staff), can help program management prioritize and managt 
improvements, concentrate resources and address the most. 
important payoff areas. Performance indicators can be put. 
into place before (e.g., evaluation framework) or after a . 


Program Evaluation 


program is “up and running”. Research into and the setting 
of competitive benchmarks and service standards is an 
important aspect of TQM in which audit and evaluation can 
support quality improvement efforts. The OCG is assisting 
both functional specialists and line managers in client 
satisfaction monitoring and performance indicator 
development through a series of guides and discussion 
papers (several of which have already been published, 

while others are in draft form). 


“Announcing the Program 
__ Evaluation Information 


à roue | 


Traditionally, audits and evaluations have been periodic, 


retroactive “spot checks” of efficiency and effectiveness. Both | service is free, except for long à distance ee he 
have the ability to be more proactive, realigning processes to | applicable. _ 

build quality in at the front end. Indeed, the emerging 

paradigm shift in internal audit focus from ex post to ex The network flees 

ante reflects the TQM tenet that quality is built in at the _ _ 

front end, not inspected in at the back. Also, witness the = ite ad interdepartmental ee data 


transfer; 

- electronic mail; 

soltware: / 
on-line on and copy of ae iter 


increasing importance of systems-under-development audit 
and new approaches to auditing management-control 
systems. Both program evaluators and internal auditors 


need to be knowledgeable about TQM efforts and initiatives 

so that their efforts are properly planned and coordinated documentation; - 

with that of departmental management. This will avoid — on-line search of the OCG/EAB Resource Centre 
duplication of effort, frustration and overburdening clients à Catalogue; 

with surveys. As internal service providers, both functions _ — special interest groups (SIGs) for a. 
may also wish to reexamine their roles, services and and exchanging information on specific topics of 
products in the light of TQM principles. interest; and 


— 24-hour-a-day access via standard telephone lines. — 
The knowledge, skills and approaches encompassed by the . 


evaluation and audit functions can provide expertise and For more information or communication software, call 
advice to line managers on how to manage better: how to the Information Technology Audit Development Centre 
achieve their program objectives more effectively and (ITADC) helpline at (613) 957-7065. 


efficiently through realigning management systems and 
measuring results; and how to seek out and report upon 
client expectations and reactions. Evaluators and auditors 
have a wealth of experience in these areas and should play a 


For access to the network, have your computer call 
(613) 957-7054. 


key role in advising on the implementation of ongoing and He Bower, 
periodic monitoring systems. Such systems are essential to : : 
Evaluation and Audit Branch 


measuring the existing quality of results and processes so 
that improvement can be demonstrated and managed. The 
review functions, with their pre-existing mindset of 
stimulating improvement, therefore can be an essential 
support to the TQM movement that is taking root in the 
federal public sector, and have a significant impact on cost- 
effective and accountable public administration. 


Office of the Comptroller General 


doser Do you have an interesting article on program evalua- 


tion that we can publish? The Newsletter welcomes all 


Ms. Fader is an evaluation analyst with the Operations ; : 
. , contributions in either official language from the eval- 


Division of the Evaluation and Audit Branch, Office of 
the Comptroller General 


uation community. For further information, please call 
the Editor, Paulette Somerville at (613) 957-7167. 
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SS ED 


Program Evaluation 


1992 Professional Development Schedule 


Seminar/Workshop 


Program Evaluation: 
Current Theory and Practice 
(in residence) 


Program Evaluation 
Orientation Workshop 
(in residence) 


Internal Audit 
Orientation Workshop 
Gin residence) 


Methods for Program 
Evaluation 


Methods for Program 
Evaluation 


To Register: 


Please contact Paulette Somerville at (613) 957-7167. Workshops may be cancelled three weeks prior to scheduled presenta- 


ETAT LSS ST DPE OE ARIE LEY 


Office of the Comptroller General 


Evaluation and Audit Branch 


Language Date 

French May 24-29/92 

English Sept. 20-25/92 

English Oct. 4-9/92 
(tentative) 

French Oct. 26-30/92 

English Nov. 23-27/92 


tion should there be insufficient registration. 


Program Evaluation Orientation 
Workshop (English) 

Program Evaluation: Current 
Theory and Practice (French) 


These five-day in-residence workshops 
provide an overview of the phases of the 
evaluation process. By means of case 
studies, participants will have the oppor- 
tunty to complete the steps in the evalua- 


tion process and receive feedback on their 


efforts from their colleagues and session 
leaders. All newly-appointed evaluation 
officers are encouraged to participate in 
these core workshops. The material cov- 
ered is considered to be preparatory to 
other EAB seminars and workshops. 


Internal Audit Orientation 
Workshop 


This five-day in-residence workshop pro- 
vides new internal auditors with a hands- 
on approach to auditing in the federal gov- 
ernment. As such they will be exposed to 


both the methodology, process and 
professional practices currently used in 
auditing the efficiency and effectiveness 
of the management framework of a 
department. This will be done through the 
use of lectures, class discussion, small 
group assignments and participation, as a 
member of a syndicate, in a live case 
study. This interactive workshop will 
encourage participants to learn by doing 
in a team-building environment where 
communication and interpersonal skills 
will be cultivated. 


The workshop is directed at newly 
appointed internal auditors, who have 
either little or no previous experience 
with internal auditing in the Government 
of Canada. A knowledge of the basic 
material contained in this workshop is 
considered necessary for participation in 
more advanced internal audit workshops. 


Location 


Le Tremblant Club 
Mont Tremblant, Quebec 


Donald Gordon Centre — 
421 Union Street 
Kingston, Ontario — 


Transport Canada 
Training Institute 

Cornwall, Ontario 
Place du Portage 


Phase III - Room 511 
Hull, Quebec 


Govt. Conference Centre 
2 Rideau Street, Ottawa 


Methods for Program Evaluation 


This five-day workshop provides evalua- | 
tion practitioners with a review of the 
advantages, disadvantages and appropri- | 
ateness of a range of methods available 
for determining program outcomes. In 
addition to attending lectures, participant! 
engage in problem-solving workshops and 
make presentations on the results of these 
exercises. Experienced program | 
evaluators will present examples illustra- 

ting what methods and evaluation designs 
were used in actual, completed program : 
evaluation studies and will discuss how | 
these fared. 


This workshop is aimed at evaluators who: 
have some basic evaluation experience or | 
who have attended the OCG Orientation 
Workshops (English or French) and who 
wish to refresh or improve their 
knowledge of program evaluation | 
methods. 
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The OCG Mission 


We are all experiencing a period of 
unprecedented change in the public 
sector environment. The need for 
change has been stimulated by the 
challenge to government of providing 
the high quality services demanded by 
Canadians while current economic 
reality dictates that resources are more 
scarce than ever. 


Many government departments have 
first responded to these demands by 
making sure that what they are doing is 
consistent with objectives that are 
appropriate for the times. We at the 
OCG are no different, and we have 
recently developed a statement of our 
Mission, supported by strategic 
objectives and the values which we hold 
that will guide our work. 


I want to take this opportunity to 
describe our Mission and the motto for 
you because it affects the way the 
Evaluation and Audit Branch 
undertakes its work. We believe that 
the most important aspect of that work 
is how we support line managers, 
primarily by exercising a leadership 
role within the program evaluation and 


Doug Wood, Deputy Comptroller General, 
Evaluation and Audit Branch, OCG. 


internal audit communities but also by 
working directly with managers in such 
areas as performance measurement. 


The OCG Mission is: 


We are leaders in innovative 
approaches to management processes 
for improved program delivery. 


| 


Program Evaluation | 


At first, this statement might seem to be somewhat general; 
it doesn’t say anything about evaluation, audit or 
performance measurement. However, like all good mission 
statements, it contains the elements of why and what we see 
as our work, while remaining flexible on the methods. Let 
me break it down for you. 


We are leaders... 


In today’s environment, we recognize that central agencies 
also have a role in serving the departments and agencies 
which deliver services to the public. In serving departments 
and agencies, we in the OCG must still be sure that sound 
policies and the essential management controls are in place. 
That message must be clear to all managers. However, 
today we lead management change rather than dictate it. 
We provide policy guidance and advice based on consultation 
with those empowered managers who are responsible for 
delivering the services. In Evaluation and Audit Branch, we 
also recognize that leadership means being accountable for 
the value we add to service delivery across government. 


... in innovative approaches... 


We must innovate because meeting expectations with a 
shrinking resource base means that we have to do things 
differently; it is that simple. Fortunately, with the 
technological tools available today, we can do some re- 
engineering of our processes to improve productivity. 
However, to remain credible, we in the review communities 
must be positioned to advise management on how re- 
engineering can be applied to the activities we review. 


. .. to management processes for improved 
program delivery... 


Within the OCG Mission, we in Evaluation and Audit 
Branch are pursuing a goal of cost-effective and accountable 
public administration. We see that evaluation, audit and 
performance measurement are key management tools and 
are central elements in demonstrating the kind of 
accountability that is due Canadians. 


We recognize that the OCG (as well as the evaluation and 
audit communities) does not deliver services to the public. 
We work to help program managers deliver those services 
more efficiently and effectively by helping to provide a 
workable management framework and by suggesting 
alternative programming designs. 


| 
I would also like to share with you the supporting strategi 
objectives of the Office. There is nothing more than a sense 
of good public administration behind most of them. Yet we’ 
find that expressing and publishing them keeps them fres] 
in our minds. They help to keep us on track as we develop | 
operational responses to the many challenges and 
opportunities facing us. 


The OCG objectives are: 
¢ to reduce the cost of government administration; 


* to promote cost-effective program delivery; 


* to promote managerial commitment to control of | 
and accountability for public resources; 


Il 
1 


* to advocate and help implement innovative | 
management change; and 


| 
¢ to have capable and innovative people to respond to | 
the challenges of our changing environment. : 
Like many organizations, we are currently working to | 
develop meaningful performance indicators at the Branch | 
and Office levels to demonstrate the extent to which we are 
meeting our objectives. We hope to share the results with | 
you by the end of this year. | 


; 


| 


Finally, I want to tell you about our motto — “Your success i 
our business.” We think it sums up the way we want to woi 
with managers and, of course, with the evaluation, audit, | 
and financial communities. If ever our actions suggest 
something different, I hope that you will tell me! 


| 
Doug Woo 


| 
| 
| 
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fany readers will have seen the TV commercial by an 
nvestment brokerage firm that claims to measure success 
one customer at a time.” This is an important concept for 
nanagers and evaluators seeking to measure the 
erformance of government programs. Some progressive 
etailing strategies are based on the idea of the “total shop- 
jing experience” for the customer and not sales per clerk 
r per square foot. The idea is to look at how the product 
r service is perceived from the outside, not the inside. 


he classic strategy for performance measurement is to 
reak a program up into its component activities and 
neasure the quality of each. A (simplified) example, famil- 
ar to all, is the federal income tax program. Some compo- 
ients include publicity, forms distribution, telephone 


nquiries, processing, refund issue, collections and appeals. 


\s one might expect, managers and evaluators generate 
ind analyse such data as turnaround time for processing, 
iccuracy for telephone responses, effectiveness of collec- 
ion procedures and so on. For certain activities, it may 
so be a good idea to collect subjective satisfaction opin- 
ons from clients. Such information is useful for the 


ndividual component managers in optimizing their partic- - 


lar areas. It is less useful for accountability and optimiz- 
ng among components. 


he accompanying table gives some hypothetical service 
nd operating statistics on this example program. While 
iot perfect, these figures, taken individually, would seem 
0 indicate pretty fair performance. 


Activity 
Forms Distribution 
Phone & Letter Inquiries 
Processing 95% accurate 
Refunds 


Payments & Collections 


collection action 


Hypothetical Operating Characteristics of Income Tax Processing 


90% of taxpayers get all the forms they need (and only those) at correct address 


85% correct; 85% timely (applies to the 60% of taxpayers making inquiries) 


80% within six-week standard (applies to the 80% of taxpayers who get refunds) 


67% treated well (timely, courteous, etc.). Applies to the 20 % of taxpayers 
required to pay, of which 75% are undisputed and 25% involve appeal or 


Program Evaluation 


service ... One Client at a Time 


But let’s now look at these hypothetical numbers from the 
perspective of the taxpayer. What chance does he or she 
have of getting satisfactory total service from Revenue 
Canada, Taxation (RCT)? If we assume (for a minute) that 
the components are independent, the arithmetic cranks 
out only a 58.7 per cent chance of a satisfactory interaction 
with RCT for the typical taxpayer. In a typical household of 
two taxpayers, the chances are roughly two in three that 
someone in the household will have an account that is not 
getting the service it deserves! And that’s just in one year! 
If, instead, we assume that some of the problems are 
correlated (e.g., inaccurate phone responses lead to 
processing delays), we merely trade a satisfied client for 
one with two problems! 


Thus, a complementary performance measurement 
strategy for RCT would be to sample taxpayers’ files from 
beginning to end, documenting the problems each experi- 
enced, if any. In other words, measure the quality of the 
program one taxpayer at a time. 


This strategy would be much more meaningful for public 
accountability than the separate numbers for each compo- 
nent. Furthermore, it would permit investigations of the 
correlations between service deficiencies. The 
organization would have a much better idea of which prob- 
lems were causing the most grief to clients and, therefore, 
which ought to be solved first. 


Performance 


P ft 
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Please note that I am not recommending this rather sophis- 
ticated perspective for performance measurement where 
nothing is currently being done for the individual compo- 
nents. That is, a program is unlikely to be able to respond 
effectively to the overall measurement of client service if it 
is not already collecting more basic information on the 
performance of its components. 


Relationship to Standards Reporting 
and Accountability 


The February 1992 Budget required departments to develop 
and publicize service standards. In addition, departments 
were to report to the public on the quality of service in rela- 
tion to the standards. One common objection to this philos- 
ophy is that simple, understandable standards mask the 
complexities of the programs. A promise of service within a 
certain time frame (e.g., a refund within six weeks) is easy 
to understand. However, it may be unrealistic for certain 
clients or at certain times. The manager’s choice is to relax 
the service time commitment so that it covers more clients 
more often or to develop multiple standards for different 
clients and periods. 


But the first option allows mediocre or worse than neces- 
Sary service for many clients or periods. That is, all those 
who could have had service in six weeks or less would 
now have a standard of seven or eight weeks in order to 
expand the coverage to more difficult cases. What would 
be the optimum single standard? Fifty per cent in four 
weeks? Eighty per cent in six? Ninety per cent in twelve? 
The second option (complex or multiple standards) 
becomes more and more difficult to measure and commu- 
nicate. At the worst, it will look like a list of excuses! 


But by measuring performance one client at a time, there 
is, in effect, a customized standard for each one. The 
organization would make a commitment to treating each 
customer as he or she merits. Clients still could be 
consulted on what this means. If a representative group of 
clients were instructed on the legal, resource and other 
constraints to service, they could be very helpful in setting 
standards and even in improving systems. An organization 
would be able to report that, say, 80 per cent of clients 
were getting good service or acceptable results. 


This is simple to communicate for accountability. Anybody 
who wanted to understand the definition of good or 
acceptable would be welcome to dig into the complexities 
of definition and measurement. 


Let's try these ideas in a few other departments. Obviously, 
parallel cases can be developed for other “commercial” 
dealings such as GST collection and social transfers such 


| 
| 


as pensions. Each account should be up-to-date, accurat 
courteously served and audited. This would apply to con 
tracts with suppliers of goods, services or real estate an 
with those making the requisitions. In other words, Supp 
and Services Canada or Public Works Canada, for exam 
could measure performance on the total experience of | 
individual requests and contracts instead of (or as well a 
specific parts thereof. 


The Correctional Service of Canada (CSC) could measur’ 
its success one offender at a time. Rather than looking sé 
arately at specific programs for custody, training and cov 
selling, the department would measure whether each  ! 
offender (or each one in a sample) is getting appropriate | 
custody, training, counselling, visiting, health care, food, ! 
parole application preparation and all the other services, 
treatments and opportunities. Obviously, the opinions of! 
experts are involved in defining “appropriate” andthe __ 
“standard” will differ among inmates. It is true that certal 
deficiencies will be more serious than others (escaping ! 
and committing new crimes is worse than being required. 
to wait a few days for a course to begin). Nevertheless, b: 
measuring one “client” at a time, the CSC would be able 
to account for its custody in an integrated and understant 
able way. This example points out how departments with’ 


“helping” mandate can use this style of measurement. 


| 
Can we stretch the concept to regulation of the airline | 
industry at Transport Canada? If we take passengers as th 
focus of analysis, what are their chances of flyingina | 
plane that is properly maintained, inspected recently 
enough by a competent inspector, using properly equippe, 
and staffed airports, flown by competent and healthy cre\ 
and cabin staff, guided by competent and rested | 
controllers with proper equipment, and observing safe | 
procedures and separations? This is, in a sense, atotal | 
risk analysis. It could be cobbled together from inspection 
and maintenance reports and various operating statistics. 
However, the interrelation of risk factors would notbe . 
apparent. Why not measure the experience of an individu: 
flyer? An alternative would be to consider each flight as 
a focus. Similar measurement and reporting could be | 
applied to other regulatory programs. For example, what 
are the odds that a drug prescription is correct from the ) 
viewpoint of approval, information, manufacturing, 
pricing and abuse? 


) 


There are other types of programs where this client- or 
account-based measurement might be more difficult to _ 
apply. Could some sort of “total readiness” measurement ! 
(equipped, trained, located) apply to the “service” units | 
of departments such as National Defence, the RCMPor | 
the Department of External Affairs? Even research units | 
might want to get some overall competence or “state- 
of-the-art” rating. 


| 
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The idea is to get program managers first to measure and 
understand and then to report on the fraction of their task 
where everything is done right. Evaluators ought to be able 
i help formulate this external view of programs. This 
notion of looking at every aspect of a program, but from 

‘ customer's perspective, overcomes the objection to sim- 
plistic indicators. At the same time, an understandable 
accountability is developed through the simple reporting 
of complex standards .'"» 


| John Batchelor 
Ur. Batchelor is a senior evaluation analyst, Operations 
Division of the Evaluation and Audit Branch, OCG. 
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f you haven't heard, EARG is the Evaluation, Audit and 
review Group of Treasury Board Secretariat’s Personnel 
Policy Branch. Since 1981, its purpose has been to evaluate 
ind audit all Public Service personnel policies. Like other 
groups in the Public Service, EARG has been profoundly 
iffected by PS2000, Increased Ministerial Authority and 
Accountability (IMAA), increased delegation, and an 
hanced focus on client service. The Evaluation, Audit 
ind Review Group is moving away from the traditional 
engthy process-oriented audits and evaluations to those 
hat are more action- and results-oriented. 

| 

fast Track Evaluation and the use of focus groups have 
illowed for a much faster, more reliable and results-driven 
valuation process. Our evaluations and studies are now 
leveloped i in wider consultation with the evaluation and 
udit community, front-line managers, and personnel 
pecialists. The involvement of these stakeholders in all 
ihases of our studies has improved both efficiency and 
esults, as well as the level of satisfaction, of departments. 


a addition to its traditional responsibilities, EARG has 
ecome increasingly involved in other areas of human 
esource management. Specifically, a more significant por- 
ion of EARG’s resources and activities are concentrated 
m accountability related projects. The most recent initia- 
ves in this area include: the Shared Management Agenda 
SMA); Departmental Management Assessment (DMA); the 
luman Resource Management Accountability Framework, 
yhich includes the joint Public Service Commission/ 
treasury Board Secretariat project to develop 

derformance indicators”; Empowerment; and a review of 
lelegated Human Resource Management Authorities. 
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OCG Practicum Session 


Managing your Career: What You Can Do! 


January 5, 1993 
8:30 a.m. to 12:00 noon 


Auditorium, National Library & National Archives Bldg. 
395 Wellington Street, Ottawa 


For more information, please call the Corporate Affairs 
and Professional Development Division of the OCG _ 
at (613) 952-3337. 
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Have You Ever Heard of EARG? 


The SMA process was introduced in 1991 by the Secretary 
of the Treasury Board. It is aimed at fostering better 
management in government through clear communication 
at senior levels between the Treasury Board Secretary and 
a deputy head-departmental management team. The EARG 
played a key role in the conception and development of 
this initiative. It has also assumed a coordinating role in 
identifying key human resource management issues from 
the Personnel Policy and the Human Resources 
Development branches. These issues are included in the 
briefing notes used by the Secretary of the Treasury Board 
at his meetings with deputy heads and form the basis of 
the SMA agreement. 


As part of the annual DMA process, where the Secretary 
of the Treasury Board appraises deputy heads, the Director 
of EARG initiated an integrated approach to evaluating 
the performance of a deputy head-department in all 
aspects of “people management.” The Director established 
and chaired an inter-branch committee comprised of 
representatives from the Personnel Policy Branch (PPB), 
Human Resources Development Branch (HRDB) and 

the Official Languages Branch (OLB). The observations 

of all three branches were consolidated into an overall 
deputy head-departmental assessment and prepared by 

an EARG staff member for approval by the executive 
management teams from the PPB, HRDB, and OLB. The 
consolidated appraisals then formed part of the final 
TBS/OCG evaluation to be used by the Secretary of the 
Treasury Board at the Committee of Senior Officials 
(COSO) Review. 
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The Evaluation, Audit and Review Group has introduced 
and is chairing a steering committee of directors-policy 
specialists from each PPB division to develop a new 
approach where one portfolio manager will serve as the 
main source of information about a particular department. 
The portfolio manager will respond to inquiries, within 
TBS and OCG, related to SMA, IMAA and DMA processes 
as well as major departmental initiatives. Information on 
departments will be shared with policy specialists and with 
portfolio holders in other branches so that everyone 
involved with a given department will have a global picture 
of its operations, activities, and issues. This approach will 
ensure that the Secretary is kept abreast of current human 
resources issues in all departments. The prime objective 

is to develop a corporate knowledge of a department’s 
personnel management policies, systems and issues. The 
steering committee will soon be asking departments to 
join it in reviewing the current PPB relationship with 
departments and, if necessary, develop an improved 
approach to delivering PPB services to departments. 


The Evaluation, Audit and Review Group is also undertak- 
ing a major review of the delegation of human resource 
management authorities. In response to the PS2000 empha- 
sis on increasing delegation to front-line managers, the 
objective of the first phase of the review is to determine 
the extent to which departments have sub-delegated 
human resource management authorities. Through the use 
of the focus groups, the second phase will highlight any 
significant problems or issues that are faced by both direc- 
tors of Personnel and front-line managers. A report will be 
produced describing Public Service-wide problems and 
best practices in delegating human resource authorities 
that are most crucial to front-line managers in their day-to- 
day “people management” activities. 


In cooperation with the PSC, OCG, HRDB and Supply and 
Services Canada, EARG is also undertaking a study to 
develop performance indicators that will provide informa- 
tion essential for human resources management account- 
ability. During the course of the study, the project team 
will identify any initiatives and trends in the development 
and implementation of performance indicators in all 
departments. The study will contribute to the development 
of a new human resources management accountability 
framework. 


Jointly with the HRDB, EARG staff also contributed signifi- 
cantly to the Empowerment project. It is aimed at increas- 
ing the understanding and use of empowerment attitudes 
and practices directed towards a more results-oriented, 
responsive and accountable Public Service. As a first 
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product, the Empowerment steering committee has deve} 
oped a “Tool Kit” and a Communication Strategy on } 
Empowerment. Contributions to this project have come 
from a variety of sources, including focus groups f 
sponsored by the OCG financial community and the 
Winnipeg Regional Council, as well as a round table 
discussion sponsored by the Public Policy Forum. 


These different projects illustrate how the role and 
functions of the Evaluation, Audit and Review Group haw 
changed over the last few years. The Group has shifted 
from its traditional monitoring role to more consultative | 
and cooperative evaluations of Public Service personnel | 
policies. Our focus has also shifted to support the | 
implementation of PS2000 recommendations that will | 
result in a more efficient, accountable, and client-centred | 
Public Service. 


Should you wish to obtain more information on EARG’s | 
activities, or have an interest in one of the projects 
and would like to work with EARG, please contact | 
André Robert, Director, TBS/PPB Evaluation, Audit and | 
Review Group at (613) 952-3141. "=> | 


Simon Carsoi 


Mr. Carson is a Co-op student with the Evaluation, Audi 
and Review Group of the Personnel Policy Branch. 


. Others’ Views of Federal 
Evaluation 


_ Two reports critical of Canadian federal evaluation | 
- were published recently by the Office of the . 


Hein Development. 


Reflections on Adminisiratye Reform ds in Lie 

… Government of Canada, written by Al Johnson, 
was written as a background study for the current 
OAG audit of federal evaluation. The report 
commissioned by the CCMD Research Program 
and written by Duncan Campbell is called 
Program Management and Program PR 
The Shifting Interface. 


Copies of these reports are available at no charge - 
from the OAG Library, phone: (613) 995-1925, 
extension 6208. 


Anew special interest group (SIG) has been set up on 
_ the Audit/Program Evaluation Information Network 
(A/PEIN) for those interested in Total Quality 
Management (TQM). It will provide users with a means 
of exchanging information on theory, practice, courses, 
_ conferences, events, reference books, and departmental 
_ initiatives in the domain of TQM. A broad range of topics 
_ related to quality management are envisaged, such as — 
client satisfaction, employee morale, organizational _ 
culture and “climate,” empowerment, methodologies 


Total Quality Management on A/PEIN 
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and tools, continuous improvement, awards and recog- 
nition, benchmarking, and performance measurement. 
The SIG Administrator is Allison Fader, Analyst, 
Evaluation and Audit Branch, Office of the Comptroller _ 
General, 10th floor, 300 Laurier Ave. West, Ottawa, 
Ontario K1A 1E4, phone: 957-7230, fax: 957-7240. If you © 
wish to place documents on the SIG, please submit them _ 
to Ms. Fader in electronic (ASCII or WP) format. If you _ 
are not yet connected to the A/PEIN, please contact our 
helpline at 957-7065 for details on access. 


Evaluation of Consultation: An Introduction 


Consultation between the government and citizens in 
developing policies on delivering services is among the 
measures recommended by PS2000. Consultation is seen 
as a key element in service to the public and in making the 
Public Service more responsive to the public. It is a funda- 
mental aspect of good management practice. Program 
evaluation can provide significant support in improving 
consultation activities. 


The purpose of this article is merely to offer a very general 
framework for discussion and to invite evaluators with 
experience or interest in this area to share their comments 
in order to stimulate further reflection or to rework the 
proposed framework. Our observations are based prima- 
rily on an analysis of the available literature. 


1. Clarifying the concept 
of consultation 


The first step is to clarify what we mean by consultation. 
As an example, let us identify some of the expressions 
which, in people’s minds, mean consultation: communica- 
tion, information, informal consultation, public 
consultation, consulting the public, participation of the 
public, and so forth. There is a consider- 
able risk of confusion regarding the con- 
cept of consultation because the distinc- 
tion between the terms information, 
consultation and participation is 
often unclear. 


However, there is a fundamental 
difference in the end product of 


these activities within the government. Indeed, it is both 
the goal and the nature of the relationship between the 
public and the government that distinguish these notions 
from each other. 


Generally, when a public institution carries out 
consultation, it is within the framework of its planning 
mechanisms (town planning, transportation, environment, 
services, etc.) and its decision-making process. The aim 
of introducing consultation within the decision-making 
process is to improve the quality of the decision and make 
it more socially and politically transparent and effective. 


However, a public institution involved in a consultation 
process always retains its full responsibility and final 
decision-making authority. This is what makes consulta- 
tion different from other forms of participation by 
citizens, just as it is different from information or public 
relations. 


Let us briefly identify the differences between these 
concepts: 


¢ information, for a public 
institution, involves making its 
decisions known to the public 
and explaining them; the insti- 
tution retains full authority to 
influence the public and 

make the decision; 


¢ consultation, for a public 
institution having a decision 
to make on a subject that 
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affects various publics, involves being open to being 
influenced by those publics; the institution retains 
responsibility and final decision-making authority; 
and 


° participation, for a public institution, consists of 
involving the public in its decisions and sharing its 
decision-making authority, for example, by delegating 
authority and forming a partnership. 


2. Evaluation of consultation 


The designers of PS2000 advocate consultation between 
the government and agencies on the one hand, and the 
public on the other hand, and emphasize that it can take 
many forms, official or unofficial. In this article, we 

will examine only the so-called “formal” process since, 
in its “informal” form, consultation is quite difficult to 
analyse. 


Implementing a consultation process first requires 
developing a project or program in a way that the process 
can later be evaluated like any other program. The institu- 
tion must decide in which areas consultation would be 
appropriate and must define its needs for consultation. 
These needs are sometimes set by law or administrative 
regulation (as in the case of the former Law Reform 
Commission of Canada). 


In an orientation document on this issue, the PCO! 
suggests that consultation needs are generally at two 
levels: (a) service delivery (planning delivery of service 
and provision of goods), and (b) development or 
orientation of policy and programs. While a number of 
government departments and agencies adopted this 
method of dealing with the public even before the 
proposed reform by PS2000, very few evaluation studies 
have been carried out. 


Consultation Program 


At the outset, the institution must determine whether 
consultation is the appropriate vehicle to achieve the 
desired result. A consultation program must therefore 
include objectives, activities, results, resources and, lastly, 
an evaluation. Two of these elements are of particular 
importance from the standpoint of evaluation: objectives 
and activities. 


1 Draft of “Guidelines for a Consultation Mentality,” Privy Council Office, 1991. 
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Objectives 


The objectives of a consultation program depend on 
the nature, subject and scope of the consultation. It is ! 
essential to set clear and precise objectives before 
implementing any consultation process. One can then | 
determine the consultation activities or methods, the ! 
target public, the budget and time needed, as wellas | 
identify the results to be achieved. 


| 


Consultation activities | 
The consultation activity or method of consultation is | 
determined, as we have just seen, by the objectives. 
There is a wide range of possible activities, from creat-! 
ing consultative committees, to holding public | 
meetings, to individual meetings with groups, to | 
forming interest or working groups, to holding informa; 
tion sessions and surveying public opinion. 


Designing a consultation strategy is important in terms 
of the later evaluation of the process. The strategey 
should show a logical link between the general 
objectives, the objectives for each activity and the 
results and effects achieved by the consultation 
program. It may also help identify any problems that 
could arise. 


3. Evaluation of the 
consultation program 


Consultation can be evaluated from different perspectives: 
the process itself, its impact (results and effects) or both | 
at the same time. The questions asked will depend on the | 
perspective chosen. | 


However, these evaluation perspectives are not exclusive. 
Indeed, the information obtained from an examination of — 
each of these perspectives can be quite surprising; while | 
the evaluation of the process may show it to have been 
successful, the evaluation of the impact may show that 
results and effects were less evident and that the 
objectives were not achieved. 


Sometimes, it may be more practical for managers to 

conduct the evaluation of the process or the consultation | 

activities themselves. However, evaluating the impact 

of consultation is a more difficult undertaking because 

the aim is to measure results and effects. This requires the 

use of special evaluation techniques and the presentation — 

of evidence supporting the answers given to the | 

questions asked. | 
| 
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@ Evaluation of the consultation process 


or “self-evaluation” 


The manager of the consultation project or program 
could conduct what might be referred to as a “self- 
evaluation” of the consultation process. The purpose 
of the evaluation would be to provide the department 
and manager with information on the proper manage- 
ment of the consultation process, including the man- 
date, the planning of the process, the timing, the bud- 
get, the logistics, the nature of meetings, publicity and 
communications (public relations), the target popula- 
tion and its participation, as well as communication of 
the results of the consultation. 


Depending on the information required, the manager 
could evaluate the consultation process at each stage 
or at the end. The advantage of a stage-by-stage 
assessment is that it is possible to exercise some con- 
trol over the activities and to make adjustments when 
problems arise. It also makes the final evaluation eas- 
ier. A stage-by-stage evaluation can be useful when 
the process extends over a long period of time, is 
complex or is conducted country-wide. 


A final evaluation examines the entire process to 
determine the results of each element (for example, 
the mandate, consultation activities, timing), its 
degree of success and what lessons can be learned for 
other consultation processes. 


Whenever necessary, the manager should request 
assistance from departmental evaluators to help with 
the conceptual and technical aspects of the self- 
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evaluation. This assistance will also be very useful in 
the later evaluation of the consultation program. 


(b) Evaluation of the impact of consultation 


Evaluating the impact of consultation is intended to 
provide information on its results and real effects. The 
aim is to: (a) review the entire program (logical link 
between objectives, activities and results); (b) identify 
how and to what extent the program’s objectives were 
achieved; (c) determine the success or failure of the 
program based on the results; (d) determine the degree 
to which the program, as designed, is the most efficient, 
or cost-effective, to achieve the objectives compared 
to other possible processes; and (e) tearn lessons that 
could be useful for future consultation programs. 


Clearly, consulting is not simple and easy. The effort 
put into the consultation process may not always be 
successful. For the evaluation of consultation to be 
useful to departments and agencies, it is important to 
reach agreement on what type of information is 
required and to be prepared to communicate the 
results and “real” impact of the consultation process 
to the public consulted. 


Consideration must also be given to the reaction of the 
population consulted (and interest groups), the current 
climate in the country toward numerous consultations 
and the fact that the Canadian public is also consulted 
by provincial and municipal governments, as well as 

a variety of interest groups. Too much consultation 
creates disenchantment and a feeling of saturation 
among the population. It can also create expectations 
that are then impossible to satisfy. 


Action-Oriented Evaluation in Organizations 


Canadian Practices 


Edited by Joe Hudson, John Mayne 
and Ray Thomlison 


Action-Oriented Evaluation in Organizations is a 
study of evaluation philosophy, policy, and practice 
from a Canadian perspective, reflecting 
developments over the past 20 years. 


Intended for evaluators in either the public or the 
private sector, and for students of the evaluation pro- 
cess, this book discusses and describes effective insti- 
tutionalization, planning, design, conduct, and use of 
evaluation. Case studies clarify and illustrate abstract 
principles and concepts. 


Joe Hudson is Professor, Faculty of Social Work, 
The University of Calgary 


John Mayne is Director, Evaluation Policy, Office of 
the Comptroller General 


Ray Thomlison is Professor and Dean, Faculty of 
Social Work, The University of Calgary 


This book, published in 1992, is available through Wall 
& Emerson, Inc. of Toronto, phone: (416) 467-8685. 
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IIAS Meeting in Ottawa 


During the last half of the CES Conference in May, a group 
of some 25 international scholars met in Ottawa to discuss 
their research in the area of program evaluation. The 
group, chaired by Dr. Ray Rist of the Government 
Accounting Office in Washington and representing some 
15 countries, is the Working Group on Policy and Program 
Evaluation of the International Institute of Administrative 
Sciences (ILAS). They have been meeting annually since 
1986 and this was the first meeting in North America. 


The group presented three panels at the CES Conference, 
reflecting the current research areas Performance 
Measurement, Evaluation of Policy Instruments, and 
Intergovernmental Multi-level Governmental Evaluation. 
The Canadian members of the group are Robert 
Segsworth of Laurentian University, John Mayne of the 
Office of the Comptroller General and Joe Hudson of the 
University of Calgary. 


The group has had a number of publications: 
Ray Rist (ed.). Program Evaluation and the 
Management of Government: Patterns and 


Prospects across Eight Nations. Transaction 
Publications, 1990. 
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John Mayne 
Dr. Mayne is Director, Evaluation Policy Division of 
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The Program Evaluation 
Information Network 


The Program Evaluation Information Network 
(PEIN) is an electronic, on-line system that can link 
the entire program evaluation community of the 
federal government. The service is free, except for 
long distance charges, where applicable. 


The network offers: — 

¢ intra- and interdepartmental electronic data 
transfer; 

¢ electronic mail; 

¢ software; 

¢ on-line search and copy of evaluation 
documentation, evaluation policies, guides 
and other material; 

+ lists of completed evaluations as provided to 
the Public Accounts Committee of Parliament; 

* on-line search of the OCG/EAB Resource 
Centre Catalogue; 

¢ special interest groups (SIGs) for 
communicating and exchanging information 
on specific topics of interest; and 

e 24-hour-a-day access via standard 
telephone lines. 


For more information or communication software, 
call the Information Technology Audit Development 
Centre (ITADC) helpline at 957-7065. 


For access to the network, have your computer 
call 957-7054. 


Wayne Bower 

ITADC 

Evaluation and Audit Branch 
Office of the Comptroller General 
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Annual Conference of the 
Canadian Evaluation Society 


Program Evaluation and the 
Service-Oriented Culture 


The opinions expressed in this article are those of the 
author and do not represent the position of the Office of 
the Comptroller General or the Government of Canada. 


From May 10-13, 1992, the Canadian Evaluation Society 
(CES) held its 18th annual conference in Ottawa under 

the general theme of “service-oriented culture.” More than 
430 participants from across Canada and from a number 
of other countries (the U.S., the European Community, 
etc.) listened to some 110 speakers and moderators as the 
conference examined three central issues: the pluralism 
of the questioning conducted in the evaluation process, the 
eclecticism of its methods and the diversity of its uses. 


Pluralism of the questioning 


It is almost impossible to summarize this pluralism as it 
was revealed throughout the conference. However, there 
was one fact that clearly stood out in the discussions: 

the parameters of evaluation are sufficiently broad to 
cover each dimension of organization, from the structuring 
of operations to the redefinition of an organization’s raison 
d’être. These parameters can be adjusted to meet the 
specific needs and demands of the organization of which 

it is a part. Whether these needs are ontological (e.g., a 
review of the validity and postulates of a given program 

or practice), strategic (e.g., evaluation as a forum for 
public consultation to determine the needs of a target 
population), operational (e.g., improvement in the imple- 
mentation of a program), or logistical (e.g., seeking a 
better vehicle for program delivery), evaluation can play a 
constructive role in providing elements of information, 
analysis and solution in the decision-making process. 


The pluralism of evaluative questioning is also evidence 
that program evaluation has become part of the 
administrative culture of most public and private organiza- 
tions. Unlike the last CES conference I attended five or six 
years ago, where we were still examining the relevancy of 
the practice and its future, deliberations at the 1992 confer- 
ence revealed a strengthening of the practice, a maturation 
that is both multi-faceted and evolutive. 


pl 
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Eclecticism of the methods 


In a recent work, E. Guba and Y. Lincoln? encourage us 

to rethink the epistemological and methodological founda- 
tions of evaluation by being more eclectic in our selection 
of investigative methods. The 1992 conference was remark- 
able in this sense by providing participants with a vast 
array of practical workshops and panels on qualitative and 
quantitative methods. Indeed, so-called qualitative methods 
appeared to be in the spotlight this year — a situation we 
should applaud. Too often, the use of qualitative methods 
in program evaluation, as in other fields of applied 
research, have involved a great deal of improvisation and, 
unfortunately, very little rigour. 


Even if, as Natalie Kishchuk pointed out in her workshop, 
qualitative research in evaluation may be both inductive 
and comparative (although the comparison is often very 
weak), it must nevertheless be conducted according to 
certain rules, or the arbitrariness of the analyst could alter 
the investigation’s findings. The challenge is, therefore, in 
the conscious and deliberate effort to eliminate subjec- 
tivity, both by the evaluator and the “evaluee.” (The fact 
that the “evaluee” is an organization does not alter the 
challenge.) The precept that “one should not place all of 
one’s eggs in the same basket” is very relevant in this con- 
text. Eclecticism in methodology, embodied in the famous 
“multiple lines of evidence approach,” has now become an 
integral part of evaluation. However, care should be taken 
not to sacrifice the use of quantitative techniques where 
they are still appropriate. 


Diversity of uses 


Another indication of the maturation of the evaluative 
function was omnipresent during the 1992 conference: the 
variety of uses of program evaluation. This may be linked 
to the increasingly widespread idea that evaluation is, 
before being a product or a process, a service and that, as 
such, it can be used for different purposes by the organiza- 
tion. Many examples of the uses of evaluation were 
presented at the 1992 conference. We will mention just a 
few: evaluation as a measure of program performance; 
evaluation as a mechanism to follow up and monitor the 
implementation of strategies; evaluation as a measure of 
client satisfaction; evaluation as an integral part of the 
strategic and operational planning of an organization; 
evaluation as a tool to rationalize an organization’s 
structure and functions; evaluation as a pedagogical oppor- 
tunity and teaching tool; and evaluation as a jumping-off 
point for ethical and ontological discussions. The list goes 
on and on. There are as many ways to use the evaluative 
service as there are organizations that use it. That in itself 
is a sign of vitality. 


2 Fourth Generation Evaluation, Sage Publications, 1989. 
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Service-oriented culture or | 
servant of the culture? | 


As we mentioned earlier, the general theme of the 1992 
conference was the variety of different links between 
program evaluation and a service-oriented culture. The 
two invited speakers, Brian Marson (President of Service 
Quality BC and Comptroller General for that province) | 
and Rémi Lacasse (Vice President of Xerox Canada in 
Quebec), set the tone during the two plenary sessions. 
Although these two senior executives gained their experi-| 
ence in different sectors, they have both distinguished | 
themselves professionally in promoting total quality. Mr. | 
Marson apologized for what he called the “quality service | 
revolution.” Using an allegory highlighting an ostrich | 
(proponents of the status quo), a parakeet (the “big | 
mouths”), a tiger (embodying competitiveness and the | 
demand for total quality) and an owl (the wise evaluator), | 
Mr. Marson tried to convince the audience that adopting | 
total quality is now more than ever a matter of survival. | 
Mr. Lacasse supported this proposal using Xerox as a 
case in point. Xerox’s approach to total quality was | 
accompanied by a complete shake-up in the work culture | 
and management practices. By focusing on training, | 
measurement of satisfaction, vertical and horizontal 
communication and a thorough change in management 
“habits and customs,” Xerox succeeded, in the space of | 
a few years, in regaining the market share it had lost, | 
according to Mr. Lacasse, because of its arrogance and ! 
insensitivity to the needs of its clients. 


Both speakers concluded by emphasizing that, with the | 
globalization of economies and the growing sophistication 
of the population (who are at the same time consumers, | 
users and taxpayers), the pursuit of total quality is a cate- | 
gorical imperative for any large organization. And what | 
can we say about the evaluator’s role in all of this? Itis | 
summarized in Mr. Marson’s proposal, “Listen, listen, | 
listen, measure, measure, measure.” Does the evaluator 
become some sort of sherpa who carries on his shoulders | 
the heavy weight of measurement and analysis and who | 
assists the decision-makers in attaining the summit of total 
quality? In other words, rather than being an agent of | 
change, is the evaluator to become a servant of the organi- 
zational culture? Is this not a contradiction? It would | 
perhaps be appropriate to consider the consequences of a | 
reversal of the polarity of the terms “culture and service.” | 

| 
Talk and practice: q 
the singular and the plural | 


The storyline is just a bit too perfect for there not to be 
something wrong. While we listened politely and even 
enthusiastically to the stimulating words of the speakers 
in the plenary sessions, in the back of our minds we 


oo 


questioned the role assigned to the evaluator. How could 
we not be in agreement with the virtue of giving better ser- 
vice to the client? But as one participant mentioned in the 
workshops, who will be left to ask the “hard questions”? 
Does supporting the role of the evaluator as technician and 
interpreter of the measurement of client satisfaction not 
contradict the principles of pluralism, eclecticism and 
variety discussed earlier? It is revealing to have noted that, 
during the 1992 conference, there was a wide gap between 
the unified concept presented at the plenary sessions and 
the pluralism of evaluative practices. It is as though we 
were trying to build a monument to total quality while 
recognizing that reality remains a composite and is gener- 
ally unresponsive to a rallying cry which, more often than 
not, mimics poorly the “Be Good or Be Gone!” motto of 
the private sector. 


To be or not to be? 


The community of program evaluators appears to have 
also made another strong change in direction. A few years 
ago, there were vague desires among our ranks to form 

a professional association to control access to the profes- 
sion and thereby institute standardization of training 

and practice. Fortunately, there was no echo of such 
inclinations at the 1992 conference. For many, and 
certainly for those who support a multidisciplinary 
approach to evaluation, corporatism is a sure sign of 
professional sclerosis, a hardening of the arteries which 
nourish a practice that is constantly growing and changing. 
Obviously, we would prefer to focus on “what we are” 
rather than on “what form we take.” In my opinion, the 
profession will be better for it. 


Toward a post-modern 
administrative science? 


It is informative to note that there is a striking parallelism 
between the attributes of evaluation and those generally 
assigned to so-called “post-modern” art, architecture 

and literature. In the opinion of philosophers such as 
Michel Serres and Jean-Francois Lyotard, even science has 
not escaped this break-away movement from political, 
cultural and economic modernity. But what exactly are 

the characteristics of post-modernism? Precisely those 
discussed earlier: pluralism, eclecticism and diversity. 

In addition to these characteristics of post-modernism, 
there are others that can also be identified in evaluation: 
the blending of fields (research, evaluation, internal audit 
and management expertise are increasingly enmeshed), 
the contextual influence of collective or individual action 
(programs and policies are increasingly set on a contextual 
basis and interpreted historically), self-examination 
(practitioners who reflect on the theoretical and ideologi- 
cal foundation of their field), and so on. 


Program Evaluation 


We should ask ourselves whether program evaluation, in 
its current form — a form that is multi-faceted, polysemous 
and bursting from its original boundaries — will be the incu- 
bator of a “post-modern” administrative science that can 
fully report on contemporary organizational life, its adapt- 
ability, its fluidity and its “perpetual motion.” The current 
blossoming of evaluative practice is, in this sense, promis- 
ing. Program evaluation is increasingly better equipped to 
meet the many challenges that public administration will 
face in the decade ahead. 


Significant federal participation 


As in past years, federal participation in the conference 
was significant both in terms of its organization and 
logistics and an intellectual contribution. In organizing 
the conference, we recognize the outstanding efforts of: 
Hélène Boyer (NRC), Jennifer Birch-Jones (SC), 
Shelley Borys (EIC), Celine Cheah (OCG), Margot 
Cutcher (EC), Hervé Déry (Com), Jerzy Jarmasz 
and Kerry Johnston (SEC), Bill Graham (Com), Michel 
Messier (ISTC), Michael O’Brecht (MRC), Georges 
Teather (NRC), Normand Théberge (Agr) and Peter 
Wiebe (SEC). 


Many evaluation managers and evaluators of the federal 
government shared their thoughts and practical 
experiences (in order of presentation): Allen Gower (SC); 
Jim Armstrong and Barbara Darling (Canada 
Communication Group); Hervé Déry (Com); Mare 
Lemay and Ronald C. Corbeil (OCG); Michel Grenier 
(EC); Daniel Caron, Karl Boudreault, Marie-France 
D’Auray-Boult and André Hubert (INAC); Denise 
Roberge (Lab); Carol Motuz (Agr); Bev Holloway 
(HWC); Francine Chabot-Plante (Com); Martin Ulrich 
(OCG); Albin Tremblay and Ann Wardrop (EC); Brian 
Jonah (TC); Cameron McQueen and Toru Uno (OCG); 
Walter Zubrychy and Jon Fisher (TC); B. Bergen and 
D. Long (ND); Hélène Boyer and Georges Teather 
(NRC); Michael O’Brecht (MRC); Denis O’Rourke 
(CRTC); Celine Cheah (OCG); Howard Duncan (HWC); 
John Mayne (OCG); Alan Winberg (TBS); 

Hélene Boyer (NRC); and B. Kenny (EIC). 


I hope I have not forgotten anyone! "> 
Raymond D’Aoust 


Mr. D’Aoust is an evaluation analyst with the Operations 
Division of the Evaluation and Audit Branch, OCG. 
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Abstract 


Tracking Quality of Service in the 
Public Sector: Developing a Client 
Satisfaction Index as Part of a 
Performance Monitoring System 


by John Scarlett (DOC) 


Recognizing the increasing necessity to develop 
effective measurement instruments to monitor 
levels of public sector service, a client satisfaction 
index (CSI) should be considered when designing a 
performance monitoring system for a government 
program. Although it is not a substitute for full 
scale client satisfaction studies, a client satisfaction 
index can provide a cost-effective, reliable means 
of tracking client service satisfaction levels ofa 
program. 


Identifying the various issues that require considera- 
tion when developing a client satisfaction index, this 
paper describes four steps: defining the scope of 
the client satisfaction index, identifying what crite- 
ria are associated with client service satisfaction, 
developing performance standards and the actual 
design of a client satisfaction index. Issues such as 
the nature of public sector client relationships and 
the criteria asssociated with the formation of client 
service satisfaction are used to demarcate the more 
complex considerations that are required to develop 
and apply a client satisfaction index in the Public 
Service sector. The role of a customer satisfaction 
index is also explained within the larger context of 
a performance monitoring system. Finally, a case 
study of two programs is used to examine some of 
the practical challenges to applying a customer 
satisfaction index. 


If the subject of this paper is of interest to you, 
please contact John Scarlett, Corporate Review 
Branch, Communications Canada, at 990-4731 to 
receive a copy. 


Mr. Scarlett was awarded an honourable mention 
Sor his essay by the Canadian Evaluation Society 
in the 1991-92 Student Essay Contest. 


Heads of Audit/Heads 
of Evaluation | 
June 16-19, 1992 Meeting 


Conference Follow-up 


Last June, the Heads of Evaluation, Heads of Audit 

and representatives from the Office of the Comptroller | 
General held a meeting with the theme of “Helping | 
Departmental Managers Meet Their Objectives.” | 
The following is an edited version of the document | 
subsequently provided to all Heads of Evaluation | 
and Audit. It highlights a few key themes or issues | 
that were apparent during the course of the meeting. 


One of these issues relates to the clear evidence of a 
strong and increased requirement to capitalize on the 
initiatives of others, given a future of reduced resources. ! 
We spoke frequently of information-sharing; indeed we 
were presented with many examples of “best practices.” 
including value-added innovative actions taken to support 
managers in achieving their objectives. From the 
Evaluation and Audit Branch (EAB) perspective, we will | 
renew our efforts to facilitate information-sharing through 
our liaison officers, our electronic networks and learning | 
(resource) centre. We will also organize sessions to | 
facilitate information-sharing. 


Another issue is the current activities of the Office of 
the Auditor General (OAG). We heard from the Auditor | 
General himself that he wants to help change the public — 
perception of Public Service employees and politicians. ! 
Our successes in enhancing performance accountability — 
will be critical to this effort. | 


| 
{ 


Recently, the OAG conducted a survey of “well performing 
internal audit groups” in other government jurisdictions 
and in the private sector. It was noted that the internal 
audit groups are often positioned in the mainstream of 
management where they provide value-added services to | 
managers and serve as a training ground for potential 
senior managers. Several of our invited speakers | 
recommended this practice for both of our review | 
functions. | 
| 


Concerns also arose in terms of the current government- | 
wide audit “effectiveness.” Some of the hypotheses | 
presented highlighted the need to continue to work with | 
the OAG regarding expectations for program evaluation in! 
the Canadian federal government. 


| 
| 
| 
| 
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In several sessions, participants discussed the need for and 
practicality of having credible information available on the 
benefits achieved from federal internal audit and program 
evaluation activities, as well as their associated costs. 
Warnings were given about over-emphasizing direct cost 
savings-benefits at the expense of others that are less 
tangible but perhaps more important. However, given the 
evident differences that exist regarding the expectations 
for these functions (such as with the OAG), the current 
OAG Tier II audits and other studies looking in particular 
at federal evaluation efforts, most participants supported 
the need to be able to demonstrate our value, which is 
exactly what we expect other managers to do! The OCG 
will continue to work with departments to gather and 
provide such information from a corporate perspective. 


All of these considerations suggest there is a continuing 
need for both departmental internal audit and program 
evaluation units, and the EAB, to establish or maintain a 
close relationship with the officials of the OAG. We will 
continue the EAB meetings with the OAG audit staffs 
and soon get departmental Heads of our functions 

more involved. 


“People” aspects of our work were a frequent topic of 
discussion and another area for follow-up. Several sessions 
touched on the nature and extent of the skills available, 
now and in the future, to do the work. The Conference 
Board in its “Employability Skills Profile: What are 
Employers Looking For?” (1992) has outlined a set of 
academic, personal management and teamwork skills 
required for the future. The IIA’s study on “common body 
of knowledge” seems to suggest the requirements for 
certification may be enhanced. The findings of another 
study, ITA Research Study on Internal Audit’s Contribution 
to the Management of Quality/Productivity indicate there 
is aneed to upgrade the capabilities of internal auditors 
with respect to their understanding of management. 


For a number of years, the evaluation community has been 
seen as being made up of two basic types of people: “core” 
evaluators (who seek to spend a good part of their career 
in evaluation) and others for whom evaluation is a stage in 
their career as Public Service employees. The private 
sector and, increasingly, the federal government follow the 
practice of having people from the line area spend time in 
the audit function. We were given a few examples of this, 
and we must renew our efforts in this area. At the same 
time, we must make sure that there is a receptive environ- 
ment for formalizing this practice. 


Another topic discussed at various points was the relation- 
ship and linkages between the evaluator’s evaluation 
framework and the manager’s accountability framework. 


Program Evaluation 


Of particular interest were the reflections of some senior 
managers indicating that evaluation frameworks as they 
now exist are useless. We will continue to work with mem- 
bers of our communities to clarify how both evaluation 
and audit work can be made more relevant to the 
manager’s performance framework. Together we can 
determine what information about both programs and 
operations will be required on an ongoing basis, and what 
on a periodic basis. Given evolving expectations, we can 
discuss how our work can best contribute to assisting 
managers to be prepared to deal with emerging demands 
related to accountability. 


In this context speakers also indicated an increased 

need for both communities to be able to rapidly respond 
to management requests for credible information. “Just- 
in time” evaluations were mentioned. Credible, strategic 
work towards developing a performance framework 

will be useful in addressing this issue. Our communities 
(and individual units) must retain capable, experienced 
people who are best prepared to offer “expert” opinion 
on short notice. These people are a corporate resource. 
Audit and evaluation should come to be seen as contribut- 
ing more than just studies. Both should come to be seen 
as another corporate resource, in the business of creating 
“intellectual capital,” specifically by building up the 
information base on the performance of programs and 
operations. Within the EAB, we are considering the best 
means of proceeding in this area, and we hope to be 

able to discuss certain notions and proposals over the 
next while. 


On a final note, preliminary analyses of the evaluations 
suggest that the majority of attendees rated the conference 
as asuccess. This finding certainly corroborates our 
feeling that it was most valuable as a learning experience. 
Suggestions for the future included planning a shorter 
session, and giving greater consideration to holding a 
single joint session. "> 


Do you have an interesting article on program 
evaluation that we can publish? The Newsletter 
welcomes all contributions in either official language 


from the evaluation community. For further informa- 
tion, please call the Editor, Paulette Somerville, 
at (613) 957-7167. 
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Seminars/Workshops Language 


Internal Audit 
Orientation 
(in residence) 


Internal Audit French 
Orientation 
(in residence) 


English 


Program Evaluation: Current French 
Theory and Practice 
(in residence) 


Program Evaluation English 
Orientation 


(in residence) 


Program Evaluation English 


Methods 


Internal Audit Orientation Workshop 


These in-residence workshops provide new internal audi- 
tors with a hands-on approach to auditing in the federal 
government. As such they will be exposed to both the 
methodology process and professional practices currently 
used in auditing the efficiency and effectiveness of the 
management framework of a department. This will be 
done through the use of lectures, class discussion, small 
group assignments and participation, as a member of a 
syndicate, in case studies. These interactive workshops 
will encourage participants to learn by doing in a team- 
building environment where communication and interper- 
sonal skills will be cultivated. 


These workshops are directed at newly-appointed internal 
auditors, who have either little or no previous experience 
with internal auditing in the Government of Canada. A 
knowledge of the basic material contained in these work- 
shops is considered necessary for participation in more 
advanced internal audit workshops. 


Program Evaluation Orientation 
Workshop (English) 

Program Evaluation: Current Theory 
and Practice (French) 


These five-day in-residence workshops provide an 
overview of the phases of the evaluation process. By 
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1993 Winter/Spring Calendar of : | 
Professional Development Activities 


Date Location 


To register: please contact Paulette Somerville at (613) 957-7167. Workshops may be cancelled three weeks prior to 
scheduled presentation should there be insufficient registration. 


Feb. 7-12/93 Transport Canada 
Training Institute 


Cornwall, Ontario 


| 
determined | 


Week of April 4/93 Location to be 

(tentative) 

April 25-30/93 Location to be | 
determined | 

| 

May 9-14/93 Donald Gordon Centre |! 
421 Union Street 
Kingston, Ontario | 

May 17-21/93 Conference Center | 
2 Rideau Street 
Ottawa, Ontario 


means of case studies, participants will have the | 
opportunity to complete the steps in the evaluation | 
process and receive feedback on their efforts from their 
colleagues and session leaders. All newly-appointed 
evaluation officers are encouraged to participate in 

these core workshops. The material covered is considered 
to be preparatory to other EAB seminars and workshops. _ 


Methods for Program Evaluation 


This five-day workshop provides evaluation practitioners | 
with a review of the advantages, disadvantages and | 
appropriateness of a range of methods available for 
determining program outcomes. In addition to attending 
lectures, participants engage in problem-solving 
workshops and make presentations on the results of 

these exercises. Experienced program evaluators will | 
present examples illustrating what methods and evaluatior 
designs were used in actual, completed program | 
evaluation studies and will discuss how these fared. 


This workshop is aimed at evaluators who have some 
basic evaluation experience or who have attended the 
OCG Orientation Workshops (English or French) and 
who wish to refresh or improve their knowledge of 
program evaluation methods. 
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